Testing the strength of evidence (TestE)
HR&S has developed a practical strategy for assessing if outcome and impact have been reached. The strategy is based on evidence and is called; TestE (Testing the strength of Evidence).
We aim for scientifically sound evidence and to benefit from a control, randomization, quantitative statistics, qualitative probability assessment, and contribution tracing.
We compile evidence for each expected outcome and each expected impact. The strongest cases use multiple forms of evidence, some addressing the weaknesses of others. Reported effects should be plausible as outcomes of the programme activities and consistent with the strategy for change and the observations compiled as evidence should be easier to reconcile with the programme’s claims than with other possible explanations.
It is of no importance whether an outcome was achieved or not, this is plain testing of evidence with no bias. Lessons learned on lack of outcome is as important.
Monitoring data shall eventually be uploaded real-time on RISEsupport.se, a designated IT platform accessible through computers, tablets and cell phones. We offer RISEsupport.se to our partners thus enabling efficient communication around tools, strategies, and results.
HR&S claims that participatory methods contribute to creating a deeper understanding of impact. The use of peer stakeholders in the design and execution of the methodology, and the combination of a range of methods, including a questionnaire survey conducted by peer stakeholders, participatory focus group discussions, life story interviews and participatory videos, generate a wealth of knowledge on impact. It is useful to work with an expansive definition of the impact that is not confined to for example elite groups operating in isolation from the society of which they are part. Letting survey participants be involved in deliberating the concept reveals how much programme activities take place. It is also important to not uncritically celebrating programme activities, without also demonstrating the challenges that surround it. As well as documenting that creativity and endurance are part of everyday entrepreneurial activities, the survey shall also allow the people to voice the hardship and sufferings they face and to identify the areas where support is most needed. Through the use of participatory methods, the HR&S surveys provide a space for our partners together with stakeholders from the state and other key institutions to debate and challenge ideas and discuss relevant support measures. Participatory methods have many strengths, including their adaptability, flexibility, reflexivity, and contribution to learning. Despite the fact that all of the methods discussed have their limitations and are challenging to put into practice, when combined they contribute to providing a more contextualized understanding of impact compared with standard quantitative surveys. There is no contradiction between the use of standardised questionnaire surveys and participatory methodologies, and the knowledge generated by both can be usefully combined.
Change & Sustainability
The key aspects of our surveys are change and sustainability.
Concerning change we measure two aspects; i) contribution tracing; is it our programme that causes the change, and ii) outcome; do we reach Outcome? The measurement of change needs a reference. The contribution tracing measures against a control, the control can be a randomized population that did not take part in the programme.
Measurement of change can also benefit from the macro survey, how is the situation elsewhere at large, in the region, in the country, in the world? The outcome measures against the base-line, thus the situation within the population prior to the initiation of the programme.
Outcome benefits from progress markers that are identified during the planning stage of a new programme.
Concerning sustainability, we also measure two aspects; i) institutional capacity and ii) financial balance. Institutional capacity is a wide concept that covers all aspects of the programme management’s ability to maintain the programme. Financial balance measures whether the income from the programme, without donations, are more than the costs. Sustainability is with the ROPE terminology referred to as the impact.
Our evaluations shall be real-time, the design of the evaluation is done as the programme is designed, baseline data collected prior to implementation and then we benefit from all opportunities for evaluation, while implementing. During the evaluation design we agree on how to collect, compile and analyse monitoring data, when and by whom. Lessons learned are compiled and addressed real-time.
We identify the study sample and the control, prior to implementing the programme, we reflect over sample size and external validity to meet the requirements of the selected statistical method, and benefit from simple randomized evaluation, whenever possible. We base the evaluation topics on Outcome Challenges, the quantification on Progress Markers and then measure against Baseline, and if randomization against Control.
Micro data Survey design
Agree on and implement how to collect, compile and analyse monitoring data, when and by whom.
Simple randomized evaluations
Randomized evaluations are a type of impact evaluation that use a specific methodology for creating a comparison group—in particular, the methodology of random assignment. Thus, impact evaluations that are scientifically sound usually compares outcomes of those (individuals, communities, etc.) who participated in the programme against those who did not participate. They generate a statistically identical comparison group, and therefore produce the most accurate (unbiased) results. Randomized evaluations tend to produce results that are very easy to explain.
What is the appropriate level or unit of randomization?
What is the appropriate method of randomization?
How would we implement the randomization?
Study sample & control
Ensure ethical sound selections.
Study sample: Who?
Sample size: How many?
External validity: i) Ensure that a person asked is representative for the group, and that we would expect the same answer if we asked someone else. ii) Reflect over if the results of our randomized evaluations is generalizable to other contexts.
Control: Who, where, how many?
Reflections on ethically sound selections
Without denying access: It can be possible to conduct a randomized evaluation without denying access to the intervention. For example, we could randomly select people to receive encouragement to enrol without denying any interested participants access to the intervention.
Pilot-phase randomization method: An ideal time to conduct a randomized evaluation is during the pilot phase of a programme or before scaling up. During the pilot phase, the effects of a programme on a particular population are unknown. The programme itself may be new or it may be an established programme that is targeting a new population. In both cases programme heads and policymakers may wish to better understand the impact of a programme and how the programme design might be improved. Almost by definition, the pilot programme will reach only a portion of the target population, making it possible to conduct a randomized evaluation. After the pilot phase, if the programme is shown to generate impact, the aim must be that the programme is replicated or scaled up to reach the remaining target population.
Two versions of an intervention
It may be useful to compare two different versions of an intervention, such as an existing version and a version with a new component added.
Quantitative analysis – Statistical method
Basic statistics: The basic assumption to be made is that a set of data, obtained under the same conditions, has a normal or Gaussian distribution. The primary parameters used are the mean (or average) and the standard deviation, and the main tools F-test for precision, t-Tests for bias, Linear correlation and regression and Analysis of variance (ANOVA).
Simple comparison: With randomized evaluations, the simplest method is to measure the average outcome of the targeted group and compare it to the average outcome of the control group. The difference represents the programme’s impact. To determine whether this impact is statistically significant, one can test the equality of means, using a simple t-test. One of the many benefits of randomized evaluations is that the impact can be measured without advanced statistical techniques.
Propagation of errors: The final result of a Programme is calculated from several activities (outputs) performed during the implementation and the total error in a programme is an adding-up of the sub-errors made in the various steps. The bias and precision of the whole Programme are usually relevant parameters.
Qualitative assessment – probability methods
With qualitative assessments, and contrary to statistical methods, the quality of the evidence is not judged by the sample size (the number of observations) but rather the probability of observing certain pieces of evidence. Qualitative impact evaluation includes assessing the contribution made by a particular intervention in achieving one or more outcomes, commonly referred to as a ‘contribution claim’. TestE benefit from process tracing to assess our Strategy for Change and from Contribution tracing to examine the contribution by external stakeholders . We also address Team operations, Cost- benefit, Needs driven, Equal partnership and Unexpected effects.
micro data Survey questions
- Progress markers.
- Ten Actions.
- Financial outcome.
Use semi-structured questions.
- Strategy for Change
- Milestone achievements
- Cost- benefit
- Team operations
- Needs and user driven achievements
- Equal partnership achievements
- Unexpected and/or not-targeted effects
Micro data Survey manual
Develop written survey manual.
- Number of interactions, where, when by whom.
Standardised questionnaires, questionnaire survey conducted by peer stakeholders, participatory focus group discussions, testimonies, life story interviews, participatory videos and photos etc.
- Who to interact with.
- Semi-structured interview guide.
- Survey questions.
- Other issues.
Assign the data collection team
Ensure the team has access to the tools required; written copy of the survey manual, camera, recorder, notebooks ad pens, transportation means, and that each team member is comfortable with the assignments.
The team assignments include:
- Collect testimonies.
- Taking photos.
- Record videos.
- Record interviews.
Taking notes in a dedicated notebook about everything that happens.
Ensure water-tight communication with our repondents.
Interview after interview, community after community, the exercise shall progress smoothly and a lot of interesting evidence – videos, photos, documents, recordings, etc. – shall be gathered.
At the end of a data collection exercise, it will be obvious that a data collection team has learned several lessons.
Macro data Survey design
Compile & analyse base-line data
- Compiling base-line data collected from the Programme management.
- Compiling base-line data collected from the national development plan and related information.
- Compiling base-line data collected from regional and international documents.
- Do state-of-the-art research benefitting from scientific literature.
Compile & analyse monitoring data
Compile & assess monitoring data
- Programme reports
- Financial reports
- Auditing reports
- Implemented Output & Achieved Outcome
- Microdata survey results
- Macodata survey results: Broad-trend data and literature reviews.
Testing Strength of Evidence
Personal stories: Personal stories are not easily classified, categorised, calculated or analysed. More recently, software programmes are available to facilitate categorisation of story fragments, which allows for analysis of patterns that can lead to quantitative information.
Strategy for Change: A reasonable contribution causal claim can be made if: there is a reasoned Strategy for Change for the intervention, in the sense that the key assumptions behind why the intervention is expected to work, make sense, are plausible, may be supported by evidence and/or existing research, and are agreed upon by at least some of the key stakeholders. The activities of the intervention were implemented as set out in the Strategy for Change. The Strategy for Change (or key elements thereof) is supported by and confirmed by evidence of observed results and underlying assumptions, thus, the chain of expected results occurred. The Strategy for change has not been disproved. Other influencing factors have been assessed and either shown not to have made a significant contribution or their relative role in contributing to the desired result has been recognized.
Progress marker scoring
Progress markers are scored and in order to assess the hypothesis that the Expected Outcomes and Impact have been achieved, at three levels.
Score the Progress markers for each outcome by i) giving each Progress marker a running number; ii) giving the scoring of each progress marker according to the list below, iii) giving comments that adds information to and/or explains the scorings, when appropriate, and, iv) notifying if an outcome was achieved as a result of the outputs of other actors than those involved with our programme. If so this other actor and activities shall be explained in detail.
5 Excellent 90 – 100%
4 Good 70 – 90%
3 Adequate 30 – 70 %
2 Poor 10 – 30 %
1 No performance 0 – 10 %
- Scoring based on percentage supersedes scoring based on words. Thus, when a progress marker assessment can be described using a percentage, then this is preferred.
- If the number of Target partners is increasing with time they should be grouped; Group one may for example include the 100 TPs entering the programme during a certain period of time or villages are treated separately.
- Compile Outcome challenges (OC) per Target partner and Domain with their related Progress markers (PM), scoring, and scoring comments.
- Compile lessons learned from the scoring Progress marker exercise. They may contribute to the progress markers or not, they may generate new progress markers and they may offer evidence in addition to the Progress Markers.
The Lessons learned are compiled and addressed real-time.
Impact reports are shared with all partners and uploaded on our web-site.
Definitions by HR&S
Ambition The need and ambitions as expressed by the local stakeholder, the Target partner.
Outcome challenge Challenges hindering the Target partner to reach her ambitions, as expressed by the Target Partner.
Activity Activities arranged by the Programme management Partner, addressing the Outcome challenges identified by the TP and that generates a specific Output.
Expected Output The Expected Outputs are quantified results from the Activities. The PP are in control over Outputs. It can for example be the number of active participants in a certain number of workshops that lasted a certain period of time.
Input Resources required to arrange the Activities.
Expected Outcome Actions taken by the Target partners as a results of the Activities. The programme managers do not have control.
Progress markers Measurable indicators of progress or non-progress. They are linked to the expected Outcome and are categorised at Level 1, 2 and 3.
Expected Impact We define Expected Impact as Expected Outcomes that have become sustainable over time and does not require backup from the Programme to be sustainable. The expected impact is quantitatively measureable as a result of our progress markers and we are accountable for the Expected Impact. The Expected impact is measured at the time of closing the programme. We may in addition aim to measure if our impact is still sustainable some period after we have closed the programme, maybe one, two, five and event ten years after.
Possible Impact The possible Impact is often a wide and qualitative statement, something that is desired and that may or may not happen as a consequence of our interaction, and often long after we have closed the programme. We are not accountable for the possible Impact, and we can also not claim it as goal that we strategically work towards achieving. If it actually happens, then we do often not have evidence for to which extent it was actually cause as a result of our programme.
By Millicent SIFUNA, RISE Centre Kenya.
Purpose with the survey
The key aspects of our surveys are change and sustainability.
Concerning change we measure two aspects; i) contribution tracing; is it our programme that causes the change, and ii) outcome; do we reach Outcome?
The measurement of change needs a reference. The contribution tracing measures against control, the control can be a randomized population that did not take part in the programme.
Measurement of change can also benefit from the macro survey, how is the situation elsewhere at large, in the region, in the country, in the world?
The outcome measures against the base-line, thus the situation within the population prior to the initiation of the programme.
Outcome benefits from progress markers that are identified during the planning stage of a new programme.
Concerning sustainability, we also measure two aspects; i) institutional capacity and ii) financial balance.
Institutional capacity is a wide concept that covers all aspects of the programme management’s ability to maintain the programme.
Financial balance measures whether the income from the programme, without donations, are more than the costs. Sustainability is with the ROPE terminology referred to as the impact.
Purpose of the manual
The survey manual is addressed to HR&S survey management team, who must read all these sections. It aims to assist the HR&S team in preparing and implementing surveys. This survey manual clearly outlines the methodologies for managing, conducting, and reporting on HR&S surveys.
Team: Interviewer, photographer, interpreter, coordinator.
Equipment: Questionnaire, paper, pen, voice recorder, camera.
About the programme
Please insert the information about the programme.
Approach to data collection
The manual follows the subsequent steps to pass through when carrying out a survey for this programme. This survey will be used to evaluate the impact of the programme activities on the livelihoods of members of xxx. The target population will be drawn from members of xxx. We will employ random sampling in selecting a sample of respondents from the population. Both primary and secondary sources of data collection will be used. Primary data will include semi-structured questionnaires while secondary data will involve written records. The questions will have both open and closed-ended questions. The validity of the questionnaires will be determined by HR&S Sweden. Consent to collect data will be sought from xxx.
For purposes of statistical measurement, the typography of progress markers from the programme was generated prior to the start of the programme. This generated three types of information:
These will be acquired from three different sources; i) interviews with people in charge (plus a review of book records if any), ii) interview with participants, iii) observation by interviewers. The indicators used here will include: xxx
Qualitative information will be obtained through; i) interviews with people in charge and ii) participants. The sources of information will be social harmony at homes, contribution to family, and financial independence, etc.
These will be obtained through an interview with; i) the person in charge, ii) direct examination of record books and iii) personal interview with members. This include; loans are given, delinquent loans, money in the bank account (Local / Euro), etc
Please tick the boxes that apply to you.
- I AGREE to take part in this survey / I DO NOT AGREE to take part in this survey.
- I have had the study explained to me.
- I have understood all that I have read and heard and I have had it explained to me and had my questions answered satisfactorily. I understand that I can change my mind at any stage.
- I AGREE that photos are taken and posted on HR&S and Action10 websites and social media platforms with the purpose of explaining to supporters how their investments have been used and encourage new investments / I DO NOT AGREE that photos are taken.
- I AGREE that videos are taken and posted on HR&S website and social media with the purpose of explaining to supporters how their investments have been used and encourage new investments / I DO NOT AGREE that videos are taken.
- I AGREE that testimonies are taken and posted on HR&S website and social media with the purpose of explaining to supporters how their investments have been used and encourage new investments / I DO NOT AGREE that testimonies are collected.
- In case I have agreed on photos/videos/testimonies as above, I want my name to be mentioned / I DO NOT want my name to be mentioned.
- In case I have agreed on photos/videos/testimonies as above, I have selected the below to be posted: description of photo/video/testimony No 1, description of photo/video/testimony No 2, etc.
I (name of the person taking consent) have explained the study information to the participant above, and that she/he has understood the nature and purpose of the study and consents to the participation in the study and/or photos and/or videos. She/he has been given the opportunity to ask questions which have been answered satisfactorily.
Participants names, signatures & date:
Investigators names, signatures & date:
Guidelines for completing the survey forms
The person doing the interview should explain the reason for the survey in simple but clear terms. For each respondent, the interviewer must conduct the interview in ethe same way in order to minimise differences in responses. The interviewer should not be patronising. Also, a rushed interview or the interviewer’s lack of interest might affect responses. Participation in the survey is voluntarily, and the respondent can refuse to be interviewed at any time. Interviews with the respondents will be face-to-face, in the local language, using paper and pencil questionnaires. An interpreter may be required.The interviewer will read the questions and possible responses if indicated and mark the respondent’s answers on the questionnaire.
The role of the interviewer is to ask questions, answer the respondent’s queries, record answers and edit the completed questionnaire. The interviewer must always check to ensure that the respondent has understood the question by using certain techniques like neutral probing. It is the role of the interviewer to set the pace of the interview and ensure that the respondent is focused and interested.
On the other hand, the respondent should cooperate with the interviewer and follow all the instructions. She/he should listen to questions attentively without interrupting, and ask for clarification when the question is not clear.
Guidelines for completing each question
We use open-ended questions. An open-ended question is a question that is open to any answer. They are not limited to one- or two-word answer, instead, they have multiple potential responses, and they often give room for further probing by the moderator.
Part A: General information
- Age of Respondents:
Below 20 years / 21-25 years / 26-30 years / 36-40 years / 41-50 years / 51-60 years / 61-70 years / 71-80 years / above 80 years.
- Respondents level of education:
No formal education [ ] / Primary [ ] / Secondary [ ] / College [ ] / University [ ] / Others specify……….
- Current member status:
Still a member of the programme / ex-member ofthe programe.
Note: For ex members kindly ask the question in past tense where necessary.
- How long have you been a member?
Below 12 months [ ] / 1-3 years [ ] /4-6 years [ ] / Above 6 years [ ]
- Which position did you hold in the group?
Official / member / other specify…
- Have you ever obtained a loan from the programme?
Yes [ ] / No [ ]
- What did you use the loan for?
Please indicate the extent to which the programme has helped you realize improvement over the last period of time one year / two years / three years / five years. Tick your response in the appropriate answer box.
- Since I joined the programme, I have managed to improve my knowledge.
- Since I joined the programme, I have managed to improve my skills.
- Through the programme, I have been able to pay my children’s school fees.
- Since joining the programme, I have managed to increase my monthly income.
- Since joining the programme, I have been able to establish and improve income generating activities.
Part B. For the group leader
Group structure and attributes
Please indicate the officials present; name of the person, position and the time that the person has been in the office.
- What motivated the initiation of the programme?
- What motivated the formation of the programme group?
- Indicate challenges that your group has faced.
- What are you proud of that you see your group has achieved.
- How would you desctiber the group cohesiveness.
- How would you describe the group operations.
Membership & sustainability
- For how many years has this group been in existence?
- How many members did the group have when it first started working with HR&S?
- What is the current group membership?
- Are there any other people from the community seeking to become members? If there is, how many?
- Are there any people who left the group? If so, how many?
- What are the reasons for people to have left the group?
- What is the number on average that normally turns up for group meetings?
- Please indicate how your group has grown in numbers over time, for example per year.
- Do members of this group have other saving options? If yes, which ones?
Money lenders / Borrowing from friends and relatives / Borrowing from local money lender or cooperatives / Hire-purchase or lease finance / Venture capital finance / Table banking or revolving funds or investment clubs / Banks / Other saving groups / Others, Specify
- Does the group have a separate bank account? If not, why?
- Did the group experience an increase in total revenue over the past yearsof the programme?
- Please indicate in local currency how your savings have grown over the years.
To what extent have members of programme group experienced advantages with the project?
- All the money belongs to the group / Members savings are not taken away but instead used for loaning / Interest earned remains with the group / Sources of finances used for startup of the project / Others, Specify
- What is the approximate size of total assets owned as a result of the programme?
Trainings & capacity building
- Does the group benefit from any training activities as a result of this initiative?
- If yes, what skills have you learnt as a result of this and which one do you use?
- How frequent were these meetings?
- What skills learnt by other members in the group has been used?
- Have members applied any of these skills in other contexts e.g. family, individual business
Accountability & group governance
- Do you keep a record of your group finances?
- If yes, what forms/records are you using to keep track of your group finances?
Ledger / Cashbook / Logbook /Others, Specify
- What is the maximum amount one can borrow?
- What is the least amount one can borrow?
- What is the largest loan take in the last per year by any member of the group?
- Please indicate how the maximum amount of loan one can borrow has grown over years.
- Which repayment schedules do you use? Please tick all that apply.
Regular instalments / End of loan term /Others, Specify
- How easy is it for members to borrow money?
- Have members had to extend their repayment period? Please reply per year.
i) Number of members per year, ii) Value of their loans, iii) Size of extended repayment period.
- What has the loan been used for?
How difficult is it to obtain external finance?
Part C: Focus group discussions
About the participants in each focal group; as related to Section A.
The objective of this activity will be to learn how the group members use their loans. The interviewer will ask the group members what they use their loan for. She/he will generate a table with the activites and each individual will raise the hand if she/he took a loan for the purpose mentioned.
The main aim of this will be to learn the types of businesses started as a result of the programme. The interviewer will ask about the types of businesses that different individuals started since joining the programme. She/he will ask individuals to raise their hands to respond that they had started the business types. She/he will count the number of hands raised and record in the table.
- Has monthly income increased as a result of loans through this programme?
- Are individuals expanding their businesses?
This will help get a more reliable picture of the ctivities to business development/diversification and how individuals can be assisted to improve their monthly income.
The purpose is to assess the programme performance in ensuring socio-economic development of members. The interviewer will ask members to rate how the project had ensured their socio-economic development in the following order.
- Built trust and empowerment which is aimed at benefitting the society
- My saving ability has been enhanced
- I can provide adequate food for the family
- I can keep my children in school as a result of timely fees payment
- I have improved my household assets
- Since joining, I have made a number of investments
- Wealth creation is enhanced
- Health and wellness are guaranteed
- Improving housing
- I now practice modern farming
- Others specify…………
- What are your groups plan for the future?
- Any new venture?
- What are the benefits with this programme?
- How can this programme be improved?
- In which ways is the programme operating well?
- In which ways can the programme operation improve?
- Anything else that you want to share, propose or discuss?