There is no “I” in “IE”: A Ganar Impact Evaluation
Karen Towers USAID Education Team Leader LAC Bureau
October 7, 2014
Security Issues/Target Groups
A Ganar Program: $8.9 million, 8 USAID countries
•Employment •Return to School •Start Own Business
• Alternative to Gangs/Drugs
Program Expansion Allowed for IE
Guatemala
• Honduras
Timeline for Evaluation
October 2014 3 year mid-point October 2016
End of evaluation
October 2011 Start of the evaluation
Research Questions
• Question 1 (Proof of Concept): To what extent does participation and completion of the A Ganar program increase the likelihood that youth will obtain and maintain jobs, return to school, start their own business or reduce risky behavior?
• Question 2 (Use of Sports): Does the use of sports in A Ganar increase the retention rate, job insertion rate and effectiveness of the program to teach life skills, language, math, IT and other complimentary activities?
Why Experimental?
• • • • •
Evidence Base Lack of Longitundinal How can program improve? Can it be replicated? Given scarce resources, what are the models that are working? For which groups? Most cost-effective? • Other donors
Key Stakeholders / Coordination
IO
IO
IO
IO
CO / Guatemala
IO
IO
CO / Honduras
Partners of the Americas
FOG Fund. Paiz
USAID
USAID
Guatemala
Headquarters
USAID Honduras
Social Impact
CIE N
ESA
Coordination
Lessons Learned
• • • • • • • •
Build in evaluation from the design Involve IP as early as possible Communication (key people constant; IOs; feedback) Time commitment to manage Be flexible ($ in POA budget) Donor role as referee Ethics of randomization (3 spots for every 25) Working in violent communities
How will results be used?
• Baseline helpful for Targeting • Informing Policy and Programming – Effectiveness of WFD for long-term outcomes – Sports as an effective development vehicle
• National Authority & Country Dialogue
“To Win, To Earn”
“To Win, To Earn”
Modifications •Selection Process
•Hours of Instruction “”
Using the Findings
•Improved Targeting •Program Enhancements •Credibility/Evidence of Success • Workforce Development • Sport “”
Lessons Learned
•Stakeholder Involvement •Importance of Understanding • Evaluators – Program • Stakeholders - Evaluation •Open Dialogue & Willingness to Negotiate “”
A GANAR ALLIANCE IMPACT EVALUATION
Mateusz Pucilowski Deputy Director - Impact Evaluation Social Impact
Part 1: Evaluation Overview
18
Evaluation Design Research Question 1: Proof of Concept “What is the impact of A Ganar participation on…”
Final Outcomes: • Employment • Entrepreneurship • Re-Entry into Formal Education Intermediate Outcomes: • Life Skills • Employment Skills Secondary Outcomes: • Risk Behavior • Gender Norms
Evaluation Design
Research Question 2: Role of Sport
Does inclusion of a sport constitute a ‘value-add’ ?
(t = 0 months) x X (t = 9 months) X X X (t = 27 months)
? X
X
? ?
X • Randomized assignment blocked by implementing organization and sex X ?
Guatemala
X
?
QUAL APPROACH • Case Studies In-depth interviews with 5 individuals over multiple points in time (participant, their life skills and vocational training facilitators, mentor, and employer).
• Exit Interviews Short, open-ended interviews about perceptions of the importance of the sports component in relation to their overall experience in the program
22
Part 2: Evaluation Modifications
23
Proposed Evaluation Design
Question Question 12 –– Proof Role of ofSport Concept 24
Two Country Design
Guatemala City
Tegucigalpa
25
Research Design – HONDURAS
26
Research Design - GUATEMALA
Proposed Sampling Design
Treatment Eligible Applicants Program Applicants
Control
Ineligible Applicants 28
Revised Sampling Design
Treatment
Eligible Applicants
Eligible Applicants
Program Applicants
Control
Ineligible Applicants
Direct Participants
29
Part 3: Lessons Learned
30
Research in high-threat environments • Importance of communication • Unannounced spot-checks • Interview location (household vs. public)
• Tracking youth
31
Keys to successful RCT implementation • Invest in relationships • Understand stakeholder incentives • Build capacity and mutual understanding
• Be flexible • Be inflexible! • Have a strong, communicative arbitrator
