Testing the strength of evidence for impact (TestE)

Important to Note

Our Action No. ONE “Needs & User Driven” is key, also when doing surveys. We have to be careful and never mix our own objectives, whether it is the survey manager or HR&S-Action10, with that of the Target Partner.

It is crucial that the survey management team files all the un-edited raw data from the field visits. Cecilia will collect those when visiting.

The atmosphere around the surveys shall be respectful, caring and empowering towards the respondents. It shall show that we care, that we aim to solve challenges and that we are porud of and celebrate achievements.

Micro data Survey manual

A written survey manual is created.

TestE PMP parameters

  1. Ambitions of the Programme manager partner (PMP).
  2. Outcome Challenges defined by PMP.
  3. Progress Markers defined by PMP with support from survey team.
  4. Assessment of PMP intrinsic motivation.

This data is gathered from the programme management partner and/ or target partner directly, either in a IRL meeting or through phone call or WhatsApp texting, prior to the actual survey. The questions and the way they are asked are adjusted to the context of the partner. The sessions are dated and kept as raw material.

Respondents

The respondents are identified and presented with: names, title, telephone numbers, level of education, home location, and family status. Also GDPR consent is collected if the respondent’s agree.

  1. PMP
    • Team-leader.
    • Management team.
  2. Target Partners
    How are Target Partner representatives selected, randomization or not.
    • Individuals.
    • Focus groups.
  3. Customers of Target Partners
    How are customers representatives selected, randomization or not.
    • Individuals.
    • Focus groups.
  4. Other stakeholders, such as:
    • Chief of the village.
    • Family members.
    • Teachers.
    • Head masters.
    • Other.

Survey methods

  1. Number of interactions required:
  2. Individual interviews.
  3. Standardised questionnaires.
  4. Questionnaire survey conducted by peer stakeholders.
  5. Participatory focus group discussions.
  6. Testimonies.

Type of survey questions

  1. All questions related to the progress markers are asked in a manner that makes sense to the respondents.
  2. Semi-structured interview guide.
  3. Life story interviews.
  4. Participatory videos and photos etc.
  5. Other.

Survey Team Preparations

Ensure the team has access to the tools required;
written copy of the survey manual, camera, recorder, notebooks ad pens, transportation means, and that  each team member is comfortable with the assignments.

 

Think through the scientific & ethical matters

  • Control
  • Randomization
  • Statistical assessment method
    • Quantitative
    • Qualitative
  • Contribution tracing.
  • Ethics.
unsplash - question3

Survey team & visits

Who

  1. The survey team consist of: Renaud, Elisabeth, Milohum, Yawo. 
  2. The team leader is: Renaud
  3. Date of the next visit: The visits are made on Wednesdays, the market day. xxx

When

  1. Dates of coming visits:

Where

  1. Location of the visits:

How

    1. Visit arrangements:
    2. Travel arrangements:
    3. Travel costs:
    4. The sources for covering travel costs are:

On-site interactions

Ensure water-tight communication with our repondents.
Interview after interview, community after community, the exercise shall progress smoothly and a lot of interesting evidence – videos, photos, documents, recordings, etc. – shall be gathered.
At the end of a data collection exercise, it will be obvious that a data collection team has learned several lessons.

The team assignments include:

  1. Interviews.
  2. Collect testimonies.
  3. Taking photos.
  4. Record videos.
  5. Record interviews.

  6. Taking notes in a dedicated notebook about everything that happens.

Micro data Survey questions

Social impact

Sustainable social impact means reaching the social good ambitions in a sustainable manner. Chapters and examples are provided below.

Social Good _ Direct Impact

Examples

  • Products and services are provided to under-served communities.
    • Products and services are: xxx
      • Number of customers for a each product and each service: xxx
  • Target partner business owners generate profit.
    • Number of business owners: xxx
    • Size of profit: xxx

Social Good _ Indirect Impact

 

Examples

Children well-being
Children of the women who manages HR&S small-scale businesses.

Number of meals per day increased.

  • From one to two or three.
    • Number of children: xxx

Education

  • Tuition fee paid and school attendance made possible.

Women well-being

The status of the women in the families has improved as the contribute with income.

Institutional capacity

Chapter with examples are provided below.

Accounting

  • Bookkeeping of international standards; compiling all receipts and invoices, perform bank reconciliation.
  • Annual auditing.
  • Respond to management reports from the auditor.

Examples

Project management

  • Are the number of target partners included as expected? xxx.
  • Does the target partners delivers as expected? xxx.
  • Minimum quarterly meetings with management team, notes are taken and included in the annual report. xxx.

Examples

External Communication

  • Communication between PMP and HR&S.
    • Maybe monthly reflections from PMP on WhatsApp. xxx.
  • Information from target partners on their satisfaction, needs, expectations. xxx.
    • To be written down and shared.
  • Share notes from the local management meetings. xxx.

Examples

Communication & partnership

  1. Frequent, positive, transparent and supportive communication between HQ, branch and local programme manager partner.
  2. Active, creative and locally appreciated local programme manager partner.
  3. Local programme manager partner appreciated by Branch and HQ.

Annual programme and financial reports

Examples

Target partner sevices

  • Ensure continuous feed-back from Target Partners. Information about their satisfaction, needs, expectations, xxx.

Examples

TP training coaching & team building

  1. Training sessions: xxx
  2. Coaching sessions: xxx
  3. HR&S/Action10 T-shirts “women themselves have submitted a complaint, that of offering them HR&S/Action10 t-shirts that they will often wear at each meeting.”

Financial sustainability

Sustainable economy is when income, without donations, sustainably exceeds costs.

  1. When can a sustainable economy be expected ?
  2. What is the size of the turnover?
  3. Compile sources and levels of income
  4. Compile sources and levels of expenses

Examples

Sustainable business mind-set

  • Testimony: During my next visit, I will insist on the sustainability aspect because it difficult in these villages.

Does the income exceed the expenses?

  • No, lack of funds to cover the cost for PPM’s transportation with his motorbike.

Managing a capital

  1. Is the capital big enough?
    • June 2022 the answer is no.
      The PPM requests the size of the loan to increase. “He asks me to send you this message while telling me that he would like that instead of giving 30,000 to women to start their activities, he would like it to be 40,000 given due to the high cost of the goods.”

 

Microdate Survey software

HR&S uses the QUALRTICS software
Each branch may want to download their own free version.

Real-time

Our evaluations shall be real-time, the design of the evaluation is done as the programme is designed, baseline data collected prior to implementation and then we benefit from all opportunities for evaluation, while implementing. During the evaluation design we agree on  how to collect, compile and analyse monitoring data, when and by whom. Lessons learned are  compiled and addressed real-time.

We identify the study sample and the control, prior to implementing the programme, we reflect over sample size and external validity to meet the requirements of the selected statistical method, and benefit from simple randomized evaluation, whenever possible. We base the evaluation topics on Outcome Challenges, the quantification on Progress Markers and then measure against Baseline, and if randomization against Control.

unsplash - time

Micro data Survey design

Agree on and implement how to collect, compile and analyse monitoring data, when and by whom.

Simple randomized evaluations

Randomized evaluations are a type of impact evaluation that use a specific methodology for creating a comparison group—in particular, the methodology of random assignment. Thus, impact evaluations that are scientifically sound usually compares outcomes of those (individuals, communities, etc.) who participated in the programme against those who did not participate. They generate a statistically identical comparison group, and therefore produce the most accurate (unbiased) results. Randomized evaluations  tend to produce results that are very easy to explain.
What is the appropriate level or unit of randomization?
What is the appropriate method of randomization?
How would we implement the randomization?

Study sample & control

Ensure ethical sound selections.
Study sample: Who?
Sample size: How many?
External validity: i) Ensure that a person asked is representative for the group, and that we would expect the same answer if we asked someone else. ii) Reflect over if the results of our randomized evaluations is generalizable to other contexts.
Control: Who, where, how many?

Reflections on ethically sound selections

Without denying access: It can be possible to conduct a randomized evaluation without denying access to the intervention. For example, we could randomly select people to receive encouragement to enrol without denying any interested participants access to the intervention.
Pilot-phase randomization method: An ideal time to conduct a randomized evaluation is during the pilot phase of a programme or before scaling up. During the pilot phase, the effects of a programme on a particular population are unknown. The programme itself may be new or it may be an established programme that is targeting a new population. In both cases programme heads and policymakers may wish to better understand the impact of a programme and how the programme design might be improved. Almost by definition, the pilot programme will reach only a portion of the target population, making it possible to conduct a randomized evaluation. After the pilot phase, if the programme is shown to generate impact, the aim must be that the programme is replicated or scaled up to reach the remaining target population.

Two versions of an intervention

It may be useful to compare two different versions of an intervention, such as an existing version and a version with a new component added.

unsplash - results

Quantitative analysis – Statistical method

Basic statistics: The basic assumption to be made is that a set of data, obtained under the same conditions, has a normal or Gaussian distribution. The primary parameters used are the mean (or average) and the standard deviation, and the main tools F-test for precision, t-Tests for bias, Linear correlation and regression and Analysis of variance (ANOVA).
Simple comparison: With randomized evaluations, the simplest method is to measure the average outcome of the targeted group and compare it to the average outcome of the control group. The difference represents the programme’s impact. To determine whether this impact is statistically significant, one can test the equality of means, using a simple t-test. One of the many benefits of randomized evaluations is that the impact can be measured without advanced statistical techniques.
Propagation of errors: The final result of a Programme is calculated from several activities (outputs) performed during the implementation and the total error in a programme is an adding-up of the sub-errors made in the various steps. The bias and precision of the whole Programme are usually relevant parameters.

Qualitative assessment – probability methods

With qualitative assessments, and contrary to statistical methods, the quality of the evidence is not judged by the sample size (the number of observations) but rather the probability of observing certain pieces of evidence. Qualitative impact evaluation includes assessing the contribution made by a particular intervention in achieving one or more outcomes, commonly referred to as a ‘contribution claim’. TestE benefit from process tracing to assess our Strategy for Change and from Contribution tracing to examine the contribution by external stakeholders . We also address Team operations, Cost- benefit, Needs driven, Equal partnership and Unexpected effects.

micro data Survey questions

Quantitative surveys

  1. Base-line.
  2. Progress markers.
  3. Ten Actions.
  4. Financial outcome.
  5. Other.

Qualitative surveys

Use semi-structured questions.

  1. Testimonies
  2. Strategy for Change
  3. Milestone achievements
  4. Cost- benefit
  5. Team operations
  6. Needs and user driven achievements
  7. Equal partnership achievements
  8. Unexpected and/or not-targeted effects
  9. Other
unsplash - question2

Macro data Survey design

Compile & analyse base-line data

  1. Compiling base-line data collected from the Programme management.
  2. Compiling base-line data collected from the national development plan and related information.
  3. Compiling base-line data collected from regional and international documents.
  4. Do state-of-the-art research benefitting from scientific literature.

Compile & analyse monitoring data

Compile survey-related information from
broad-trend data and literature reviews.
unsplash - data collection

Compile & assess monitoring data

Compiling data

  • Baseline
  • Programme reports
  • Financial reports
  • Auditing reports
  • Implemented Output & Achieved Outcome
  • Testimonies
  • Microdata survey results
  • Macodata survey results: Broad-trend data and literature reviews.

Testing Strength of Evidence

Personal stories: Personal stories are not easily classified, categorised, calculated or analysed. More recently, software programmes are available to facilitate categorisation of story fragments, which allows for analysis of patterns that can lead to quantitative information.

Strategy for Change:  A reasonable contribution causal claim can be made if: there is a reasoned Strategy for Change for the intervention, in the sense that the key assumptions behind why the intervention is expected to work, make sense, are plausible, may be supported by evidence and/or existing research, and are agreed upon by at least some of the key stakeholders. The activities of the intervention were implemented as set out in the Strategy for Change. The Strategy for Change (or key elements thereof) is supported by and confirmed by evidence of observed results and underlying assumptions, thus, the chain of expected results occurred.  The Strategy for change has not been disproved. Other influencing factors have been assessed and either shown not to have made a significant contribution or their relative role in contributing to the desired result has been recognized.

Progress marker scoring

Progress markers are scored and in order to assess the hypothesis that the Expected Outcomes and Impact have been achieved, at three levels.

Score the Progress markers for each outcome by i) giving each Progress marker a running number; ii) giving the scoring of each progress marker according to the list below, iii) giving comments that adds information to and/or explains the scorings, when appropriate, and, iv) notifying if an outcome was achieved as a result of the outputs of other actors than those involved with our programme. If so this other actor and activities shall be explained in detail.

Scores

5                         Excellent                                    90 – 100%

4                         Good                                             70 – 90%

3                         Adequate                                   30 – 70 %

2                         Poor                                               10 – 30 %

1                         No performance                      0 – 10 %

 

  • Scoring based on percentage supersedes scoring based on words. Thus, when a progress marker assessment can be described using a percentage, then this is preferred.
  • If the number of Target partners is increasing with time they should be grouped; Group one may for example include the 100 TPs entering the programme during a certain period of time or villages are treated separately.
  • Compile Outcome challenges (OC) per Target partner and Domain with their related Progress markers (PM), scoring, and scoring comments.
  • Compile lessons learned from the scoring Progress marker exercise. They may contribute to the progress markers or not, they may generate new progress markers and they may offer evidence in addition to the Progress Markers.
unsplash - data

Lessons learned

The Lessons learned are compiled and addressed real-time.

Protection of personal data

The study follows the ethical principles of research. Your participation is voluntary and you can quit at any time. You indicate your consent to participate by answering the questions in the survey. No personally identifiable information will be shared, and all responses are handled according to the ethical and scientific guidelines. Only HR&S have access to the collected personal data, and the data is handeled according to the “general data protection regulation” (GDPR).

 

unsplash - dotted lights

Share results

Impact reports are shared with all partners and uploaded on our web-site.

unsplash - dotted lights

Example

By Millicent SIFUNA 2020?, RISE Centre Kenya.

Purpose with the survey

The key aspects of our surveys are change and sustainability.

Concerning change we measure two aspects; i) contribution tracing; is it our programme that causes the change, and ii) outcome; do we reach Outcome?
The measurement of change needs a reference.  The contribution tracing measures against control, the control can be a randomized population that did not take part in the programme.
Measurement of change can also benefit from the macro survey, how is the situation elsewhere at large, in the region, in the country, in the world? 
The outcome measures against the base-line, thus the situation within the population prior to the initiation of the programme.
Outcome benefits from progress markers that are identified during the planning stage of a new programme.

Concerning sustainability, we also measure two aspects; i) institutional capacity and ii) financial balance.
Institutional capacity is a wide concept that covers all aspects of the programme management’s ability to maintain the programme.
Financial balance measures whether the income from the programme, without donations, are more than the costs. Sustainability is with the ROPE terminology referred to as the impact.

Purpose of the manual

The survey manual is addressed to HR&S survey management team, who must read all these sections. It aims to assist the HR&S team in preparing and implementing surveys. This survey manual clearly outlines the methodologies for managing, conducting, and reporting on HR&S surveys.

Team: Interviewer, photographer, interpreter, coordinator.
Equipment: Questionnaire, paper, pen, voice recorder, camera.

About the programme

Please insert the information about the programme.

 

Approach to data collection

The manual follows the subsequent steps to pass through when carrying out a survey for this programme. This survey will be used to evaluate the impact of the programme activities on the livelihoods of members of xxx. The target population will be drawn from members of xxx. We will employ random sampling in selecting a sample of respondents from the population. Both primary and secondary sources of data collection will be used. Primary data will include semi-structured questionnaires while secondary data will involve written records.  The questions will have both open and closed-ended questions. The validity of the questionnaires will be determined by HR&S Sweden. Consent to collect data will be sought from xxx.

For purposes of statistical measurement, the typography of progress markers from the programme was generated prior to the start of the programme. This generated three types of information:
Quantitative 
These will be acquired from three different sources; i) interviews with people in charge (plus a review of book records if any), ii) interview with participants, iii) observation by interviewers. The indicators used here will include: xxx
Qualitative
Qualitative information will be obtained through; i) interviews with people in charge and ii) participants.  The sources of information will be social harmony at homes, contribution to family, and financial independence, etc.
Financial
These will be obtained through an interview with; i) the person in charge, ii) direct examination of record books and iii) personal interview with members. This include; loans are given, delinquent loans, money in the bank account (Local / Euro), etc

Consent

Please tick the boxes that apply to you.

  • I AGREE to take part in this survey / I DO NOT AGREE to take part in this survey.
  • I have had the study explained to me.
  • I have understood all that I have read and heard and I have had it explained to me and had my questions answered satisfactorily. I understand that I can change my mind at any stage.
  • I AGREE that photos are taken and posted on HR&S and Action10 websites and social media platforms with the purpose of explaining to supporters how their investments have been used and encourage new investments  / I DO NOT AGREE that photos are taken.
  • I AGREE that videos are taken and posted on HR&S website and social media with the purpose of explaining to supporters how their investments have been used and encourage new investments  / I DO NOT AGREE that videos are taken.
  • I AGREE that testimonies are taken and posted on HR&S website and social media with the purpose of explaining to supporters how their investments have been used and encourage new investments  / I DO NOT AGREE that testimonies are collected.
  • In case I have agreed on photos/videos/testimonies as above, I want my name to be mentioned / I DO NOT want my name to be mentioned.
  • In case I have agreed on photos/videos/testimonies as above, I have selected the below to be posted: description of photo/video/testimony No 1, description of photo/video/testimony No 2, etc.

 I (name of the person taking consent) have explained the study information to the participant above, and that she/he has understood the nature and purpose of the study and consents to the participation in the study and/or photos and/or videos. She/he has been given the opportunity to ask questions which have been answered satisfactorily.

Participants names, signatures & date:      

Investigators names, signatures & date:

Guidelines for completing the survey forms

General guidelines

The person doing the interview should explain the reason for the survey in simple but clear terms. For each respondent, the interviewer must conduct the interview in ethe same way in order to minimise differences in responses. The interviewer should not be patronising. Also, a rushed interview or the interviewer’s lack of interest might affect responses.  Participation in the survey is voluntarily, and the respondent can refuse to be interviewed at any time. Interviews with the respondents will be face-to-face, in the local language, using paper and pencil questionnaires. An interpreter may be required.The interviewer will read the questions and possible responses if indicated and mark the respondent’s answers on the questionnaire.

The role of the interviewer is to ask questions, answer the respondent’s queries, record answers and edit the completed questionnaire. The interviewer must always check to ensure that the respondent has understood the question by using certain techniques like neutral probing. It is the role of the interviewer to set the pace of the interview and ensure that the respondent is focused and interested.

On the other hand, the respondent should cooperate with the interviewer and follow all the instructions. She/he should listen to questions attentively without interrupting, and ask for clarification when the question is not clear.

Guidelines for completing each question

We use open-ended questions. An open-ended question is a question that is open to any answer. They are not limited to one- or two-word answer, instead, they have multiple potential responses, and they often give room for further probing by the moderator.

Questionnaire

Part A: General information

  1. Age of Respondents:
    Below 20 years / 21-25 years / 26-30 years / 36-40 years / 41-50 years / 51-60 years / 61-70 years / 71-80 years / above 80 years.
  2. Respondents level of education:
    No formal education [ ] / Primary [ ] / Secondary [ ] / College [ ] / University [ ] / Others specify……….
  3. Current member status:
    Still a member of the programme / ex-member ofthe programe.
    Note: For ex members kindly ask the question in past tense where necessary.
  4. How long have you been a member?
    Below 12 months [ ] / 1-3 years  [ ] /4-6 years  [ ] / Above 6 years  [ ]
  5. Which position did you hold in the group?
    Official / member / other specify…

Socio-economic empowerment

  1. Have you ever obtained a loan from the programme? 
    Yes [ ]  / No [ ]
  2. What did you use the loan for?

Livelihoods

Please indicate the extent to which the programme has helped you realize improvement over the last period of time one year / two years / three years / five years. Tick your response in the appropriate answer box.

Education

  1. Since I joined the programme, I have managed to improve my knowledge.
  2. Since I joined the programme, I have managed to improve my skills.
  3. Through the programme, I have been able to pay my children’s school fees.

Income

  1. Since joining the programme, I have managed to increase my monthly income.
  2. Since joining the programme, I have been able to establish and improve income generating activities.

Part B. For the group leader

Group structure and attributes
Please indicate the officials present; name of the person, position and the time that the person has been in the office.

  1. What motivated the initiation of the programme?
  2. What motivated the formation of the programme group?
  3. Indicate challenges that your group has faced.
  4. What are you proud of that you see your group has achieved.
  5. How would you desctiber the group cohesiveness.
  6. How would you describe the group operations.

Membership & sustainability

  1. For how many years has this group been in existence?
  2. How many members did the group have when it first started working with HR&S?
  3. What is the current group membership?
  4. Are there any other people from the community seeking to become members? If there is, how many?
  5. Are there any people who left the group? If so, how many?
  6. What are the reasons for people to have left the group?
  7. What is the number on average that normally turns up for group meetings?
  8. Please indicate how your group has grown in numbers over time, for example per year.

Group Savings

  1. Do members of this group have other saving options? If yes, which ones?
    Money lenders / Borrowing from friends and relatives / Borrowing from local money lender or cooperatives / Hire-purchase or lease finance / Venture capital finance / Table banking or revolving funds or investment clubs / Banks / Other saving groups / Others, Specify
  2. Does the group have a separate bank account? If not, why?
  3. Did the group experience an increase in total revenue over the past yearsof the programme?
  4. Please indicate in local currency how your savings have grown over the years.

To what extent have members of programme group experienced advantages with the project?

  1. All the money belongs to the group / Members savings are not taken away but instead used for loaning / Interest earned remains with the group / Sources of finances used for startup of the project / Others, Specify
  2. What is the approximate size of total assets owned as a result of the programme?

Trainings & capacity building

  1. Does the group benefit from any training activities as a result of this initiative?
  2. If yes, what skills have you learnt as a result of this and which one do you use?
  3. How frequent were these meetings?
  4. What skills learnt by other members in the group has been used?
  5. Have members applied any of these skills in other contexts e.g. family, individual business

Accountability & group governance

  1. Do you keep a record of your group finances?
  2. If yes, what forms/records are you using to keep track of your group finances?
    Ledger / Cashbook / Logbook /Others, Specify

Loans

  1. What is the maximum amount one can borrow?
  2. What is the least amount one can borrow?
  3. What is the largest loan take in the last per year by any member of the group?
  4. Please indicate how the maximum amount of loan one can borrow has grown over years.
  5. Which repayment schedules do you use? Please tick all that apply.
    Regular instalments / End of loan term /Others, Specify
  6. How easy is it for members to borrow money?
  7. Have members had to extend their repayment period? Please reply per year.
    i) Number of members per year, ii) Value of their loans, iii) Size of extended repayment period.
  8. What has the loan been used for?

  9. How difficult is it to obtain external finance?

Part C: Focus group discussions

About the participants in each focal group;  as related to Section A.

Loan use

The objective of this activity will be to learn how the group members use their loans. The interviewer will ask the group members what they use their loan for. She/he will generate a table with the activites and each individual will raise the hand if she/he took a loan for the purpose mentioned.

Business development

The main aim of this will be to learn the types of businesses started as a result of the programme. The interviewer will ask about the types of businesses that different individuals started since joining the programme. She/he will ask individuals to raise their hands to respond that they had started the business types. She/he will count the number of hands raised and record in the table.

Guiding questions

  1. Has monthly income increased as a result of loans through this programme?
  2. Are individuals expanding their businesses?

This will help get a more reliable picture of the ctivities to business development/diversification and how individuals can be assisted to improve their monthly income.

Socio-economic empowerment

The purpose  is to assess the programme performance in ensuring socio-economic development of members. The interviewer will ask members to rate how the project had ensured their socio-economic development in the following order.

  1. Built trust and empowerment which is aimed at benefitting the society
  2. My saving ability has been enhanced
  3. I can provide adequate food for the family
  4. I can keep my children in school as a result of timely fees payment
  5. I have improved my household assets
  6. Since joining, I have made a number of investments
  7. Wealth creation is enhanced
  8. Health and wellness are guaranteed
  9. Improving housing
  10. I now practice modern farming
  11. Others specify…………

Closing remarks

  1. What are your groups plan for the future?
  2. Any new venture?
  3. What are the benefits with this programme?
  4. How can this programme be improved?
  5. In which ways is the programme operating well?
  6. In which ways can the programme operation improve?
  7. Anything else that you want to share, propose or discuss?