Week 3: Evaluation Types - Needs Assessments, Program Theory, and Program Process. Bob Barcelona, Ph.D. HEHD 804

Week 3: Evaluation Types - Needs Assessments, Program Theory, and Program Process Bob Barcelona, Ph.D. HEHD 804 March 26, 2013 Where We’ve Been, Whe...
Author: Egbert Townsend
1 downloads 0 Views 2MB Size
Week 3: Evaluation Types - Needs Assessments, Program Theory, and Program Process Bob Barcelona, Ph.D. HEHD 804 March 26, 2013

Where We’ve Been, Where We’re Going Where We’ve Been

Where We’re Going

 Defined program evaluation and discussed its consequences  Talked about balancing rigor and practicality  Talked about project problems and purposes  Considered the context in which evaluation takes place and the role of stakeholders  Talked about evaluation research questions and literature reviews

 Talking about evaluation types (Needs Assessments, Evaluability/Program Theory, Program Process)  Talk about populations, sampling, and measurement  Start thinking about (and writing) Chapter 3 (Methodology) – due on 4/9  Writing your Research Forum Abstract – due on 4/2!

Research Forum Abstracts • See assignment description on Blackboard for details – these are really important to adhere to! • Include the following in your abstract: – Working title for your project – The problem and purpose statement – Guiding evaluation/research questions – Basic description of the literature and/or key studies informing your project – Description of population/subjects, including pertinent characteristics – Description of the proposed research methodology (this can be really basic at this point – surveys, interviews, observations…) – A brief discussion of the implications for practice or what can be learned from the results of your study

Positive Youth Development and High Yield Natural Outdoor Experiences: Family Perceptions and Family Inclusive Opportunities Engagement in natural outdoor activities has been found to promote youth cognitive functioning, well-being, and overall physiological and psychological health. Youth are partaking in fewer natural experiences and efforts to promote participation in high-yield immersion outdoor experiences lack family inclusiveness, an essential element in positive youth development practices. The purpose of this study is to conduct a qualitative evaluation of families of youth ages 5-15 to investigate parental or primary adult caregiver perceptions of the benefits and barriers to participation in natural outdoor experiences. In addition, a qualitative investigation will evaluate three renowned outdoor organizations and their efforts to include families in high yield, natural outdoor experiences. Five guiding questions will shape the implementation and design of the study: 1) What are parents' perceived benefits of participation in high yield, natural outdoor experiences?; 2) What are parents' perceived barriers or constraints to participation in high yield, natural outdoor experiences?; 3) Do barriers or constraints to partaking in high yield, natural outdoor experiences preclude the benefits gained from participation?; 4) Do outdoor organizations aim to make the benefits of participation as identified by parents attainable through family inclusive, high yield, natural outdoor experiences?; 5) Do outdoor organizations aim to overcome barriers or constraints as identified by parents to make participation in family inclusive, high yield, natural outdoor experiences possible? Assessment of the resulting data will help ascertain if there is thematic overlap in perceptions of the value of natural outdoor experiences among parents and adult caregivers. The prevalence of family inclusive, high yield, natural outdoor opportunities made available by organizations considered leaders in the field of outdoor/adventure programming, will provide insight of potential programming needs and transferable efforts within the field.

Operation Phoenix: Improving Resiliency Factors of Institutionalized Youth in Emergent and Third World Orphanages Problem: As social violence, child abuse, and general instability increase across the globe, so does the need for emergent and third world institutionalized childcare in the form of orphanages. What does this mean for youth raised in an underfunded institutionalized setting, where research shows institutionalization negatively impacts aspects of child development and functioning (Tulviste, 2010)? It means negative youth outcomes. These youth often return to the streets focused on surviving, not thriving, and cyclically contribute to social violence, child abuse, and general instability across the globe. Due to lack of funding, there are few intentional programs promoting resiliency in this demographic. Purpose: This mixed methods study intends to measure individual, relational and contextual resiliency factors in institutionalized youth before and after a 10 week group based intervention named Operation Phoenix and modeled after Family Solutions for Youth at Risk (Quinn, 2004). If a positive correlation is found between youth resilience scores and the 10-week group intervention, a model of group intervention promoting positive youth outcomes for institutionalized youth across cultures can be developed, focusing on third world and emergent orphanages with limited funding. Subjects: 27 institutionalized youth ages 14 and above placed at the Davis Lar Orphanage in Fortaleza, Brazil, due to abuse, exploitation, abandonment or neglect are currently participating in the study. They have been separated into two groups based upon gender, male and female. Methodology: Quantitative analysis will be accomplished using the Child and Youth Resilience Measure (CYRM)-28, a 28 item Likert scale questionnaire developed by the Resilience Research Centre (2009). Concurrently, qualitative interviews using catalyst questioning, and subsequent coding, will enable researchers to hear the first hand opinions of these youth, validating outcomes of the CYRM. Questions: This study acknowledges the lack of systematic empirical studies assessing the improvement of resiliency factors in emergent and third-world institutionalized youth, and intends to answer the following questions: Will directed group interaction promoting positive youth development and life skills improve individual, relational and contextual resiliency factors in institutionalized youth in emergent and third world orphanages? Can a model be designed that will improve resiliency factors in institutionalized youth across cultures?

Different Evaluation Questions Implementation Activities

Resources Needed AS funding Afterschool program leaders Embedded curriculum School facility space

Activities Offered Daily ASP focusing on homework, academic enrichment, phys activity Family programs held at the school

Antecedent Conditions

Intended Outcomes

Outputs

% of youth who attend the ASP daily

% youth receiving daily moderatevigorous activity % of families who attend family nights

Proximal Changes Improve academic efficacy

Increase school connection Improve fitness levels Increase family attitudes

Distal Changes Improved grades in core subjects Improve district graduation rate Decrease youth obesity rates Increase family engagement in school activities

Evaluation Type

Examples

Measurement Tools

Needs Assessment

Social conditions (e.g. economy, single parent households, educational achievement, crime rates, health disparities)

Secondary data sources (e.g. census, police/juvenile justice, schools/grades, CDC data)

Needs for specific programs or facilities

Past program participation data, stakeholder surveys, focus groups, interviews

Evaluability/Logic Models

Research-based program development

Integrative literature reviews, stakeholder focus groups

Process/Quality/Outputs

Program fidelity

Observations, staff interviews/focus groups

Participation (scope, access)

Program/facility counts

Program/facility satisfaction

User surveys, interviews, focus groups

Why Needs Assessment? • So – why are needs assessments important, anyway? • What are some of the consequences of a poor understanding of constituent (or community) need? • Do you have experience doing needs assessments?

Justifying Nature & Extent of Needs • Secondary data estimates (e.g. US Census, Kids Count) • Social indicators (e.g. income, age, race, sex, education, health status) • Professional Standards (e.g. ACA, NRPA, NAA-NIOST, ACSM) • Social research: – Agency records (e.g. program participation, economic impact) – Key informant surveys and interviews – Planning techniques (e.g. Community Meetings/Public Input Sessions, Nominal Group Technique) – Population surveys

Secondary Data Analysis • Over 400 confirmed cases of child abuse and neglect in SC every week or 1 out of every 80 children 0-16 years of age • SC ranks 50th in preventive services for abused and neglected children • SC is 45th out of 50 in national obesity rankings • Over 54% of SC children do not participate in afterschool sports or lessons • 31.5% of SC high school students are overweight or obese (48% of all rural AA children) • 62% of SC children do not meet the CDC’s recommended PA levels • 9% of SC middle school students and 11% of SC high school students reported to have thought about attempting suicide in the past year • SC ranks 10th in the nation in new HIV cases annually and 7% of high school students in SC reported that they had an STD Data used in the 2012 Aiken Youth Empowerment Grant application (Brown and Barcelona, 2012)

Secondary Data Analysis • • • • • •

North Aiken has a 96.92% poverty index NAES = 92% F/R lunch rate, 89% minority, below average rating NAES = 20% failure rate in math, 27.3% in reading 92 OOS suspensions; 75% of referrals were AA males 66% of students at NAES live in single parent or foster homes Juvenile violence rates increased 129% from previous year (116 assault and battery, 77 cases of disturbing school, 6 cases of lynching) • 200 confirmed gang members operating in Aiken • 51.4% of Aiken’s youth smoke • Only 15% of Aiken high school youth ate according to USDA guidelines Data used in the 2012 Aiken Youth Empowerment Grant application (Brown and Barcelona, 2012)

• • • • • • • • • • • • •

Centers for Disease Control & Prevention (CDC) SC Dept. of Education Fiscal Year 2010 Poverty Index Aiken County News (Newspaper) 2011 SC School Report Cards 2011 SC Measurement of Academic Performance (MAP) Aiken County Juvenile Justice statistics Aiken Dept. of Public Safety Gang Survey (2006) South Carolina Dept. of Social Services Southern Institute on Families and Children SC Dept. of Health and Environmental Control SC Kids Count data SC Youth Risk Behavioral Surveillance Survey Kaiser Family Foundation State Health Facts

Agency Records % of violent crimes committed by hour of day while school is in session All Juveniles 14% 12% 10% 8% 6% 4% 2% 0% 7am 9am 11am 1pm 3pm 5pm 7pm 9pm 11pm 1am 3am 5am Source: Sickmund, M., Snyder, H. N. and Poe-Yamagata, E. (1997). Juvenile Offenders and Victims: 1997 Update on Violence. Washington, DC: National Center for Juvenile Justice

Agency Records

From Greenville (SC) Hope VI Report: A Review of the Revitalization of the Jesse Jackson Townhomes Community (Barcelona, Quinn, & Post, 2009)

Imagine that you have $100 to give to Rock Hill PRT. Where would you •designate 10% = Parks it and toOpen go?Space • • • •

17% = Recreation Programs 22% = Athletic Fields/Sports Complexes 39% = Greenways, Trails, Complete Streets 13% = Indoor Recreation Facilities Data from City of Rock Hill, SC PRT Needs Assessment, May 2011)

Survey Data (I-P Analysis) Amenity

Importance

Satisfaction

Gap

Parks and Open Space

97%

78%

-19%

Recreation Programs

96%

66%

-30%

Athletic Fields/Sports Complexes Greenways, Trails, Complete Streets Indoor Recreation Facilities

93%

77%

-16%

90%

34%

-56%

88%

27%

-61%

(Data from City of Rock Hill, SC PRT Needs Assessment, May 2011)

Survey Data Imagine that you have $100 to give to Rock Hill PRT. Where would you designate it to go? • 10% = Parks and Open Space • 17% = Recreation Programs • 22% = Athletic Fields/Sports Complexes • 39% = Greenways, Trails, Complete Streets • 13% = Indoor Recreation Facilities (Data from City of Rock Hill, SC PRT Needs Assessment, May 2011)

Survey Data

Focus Groups: “What If…” Sample Question: • As a group, answer the following question on the big notecard… “What if we could redesign and rebuild [your community] PRT from the ground up. What would that look like, and what would we prioritize?” Be specific. Think in terms of facilities, programs, amenities, organizational capacity, funding…. Don’t be afraid to think outside the box! • Locate your ideas on the map provided!

Target Populations • • Population – the universe of potential participants in an evaluation study; the group that you want to draw inferences about • Need to be able to define and describe the target population • Ex: All elementary school students attending Title I schools in Anderson, SC District 4

Important factors to consider when considering a population:  Identifiability  Access  Size

Some Sampling Approaches Sampling Procedure

Example

Population sampling

Sample all 1000 students in a middle school

Random (equal chance of selection)

Do a random sort of 400 middle school students from the overall population of 1200 students in a school

Systematic

Sample every 3rd middle school student registration record in the population of 1200 students

Stratified

Sample grades 6th, 7th, and 8th – randomly sample students by grade in proportion to the overall school population

Convenience

Sample only those middle school students who you can easily and readily access (your own classroom and the classrooms of our good friends) Consider both single-stage and multi-stage sampling!

Partial Table (full table is on Blackboard) Pop

Sample

Pop

Sample

Pop

Sample

Survey Design/Development SOME PRACTICAL TIPS: TWO BIG THINGS: • Validity = does the survey 1. Use surveys that have already been developed and are readily available measure what it is 2. Use surveys that have track records supposed to measure? of validity and reliability

– Internal (strength of 3. Adapt already developed surveys to evaluation design and your own project by making small confidence in results) modifications where necessary – External (generalizability) 4. Develop your own survey based on

the literature and in consultation • Reliability = does the with experts or key stakeholders survey produce consistent 5. Content validate your new survey results? by a panel of experts or key stakeholders

Evaluability Assessment • Why programs exist, what they need to be successful, what they do and who they serve, what outcomes they intend to achieve, and by when? • Provide a platform for measurement and evaluation! • Outline a logical causal chain – elements of the program are rationally linked together • Major components – RATIONALITY, LINKABILITY and MEASURABILITY!

Evaluability Implementation Activities

Resources Needed AS funding Afterschool program leaders Embedded curriculum School facility space

Activities Offered Daily ASP focusing on homework, academic enrichment, phys activity Family programs held at the school

Antecedent Conditions

Intended Outcomes

Outputs

% of youth who attend the ASP daily

% youth receiving daily moderatevigorous activity % of families who attend family nights

Proximal Changes Improve academic efficacy

Increase school connection Improve fitness levels Increase family attitudes

Distal Changes Improved grades in core subjects Improve district graduation rate Decrease youth obesity rates Increase family engagement in school activities

Dwyer et al. (2003) – A Useful Primer • Literature reviews help in the development of logic models/program theory – i.e. effective interventions to impact physical activity

• Community Needs Assessment of Key Informants – i.e. how should physical activity be increased among children? – Surveys and focus groups of: Parks and Recreation, Childrens’ Services, Public Health, YMCA, Public School Board, Catholic School Board, Housing Authority, other community agencies (i.e. BCGC)

• Development of draft logic model – i.e. MSAPAP – multi-strategy approach to physical activity promotion

• Stakeholder consultation – Workshops, focus groups, community meetings

Process Evaluation • The systematic and continual documentation of key aspects of program performance that assess whether the program is operating as intended • Usually focused on two kinds of questions that speak to FIDELITY: – Is the program reaching the appropriate target population? – Are the service delivery and supports consistent with intended program design? • This is an assessment of the implementation activities component of the logic model!

GoalPOST Process Evaluation Before the Program

During the Program

After the Program

Pre student survey

Monthly teacher ratings

Post student survey

Pre parent survey

Goal setting intervention

Post parent survey

Activity ratings Advisory board meetings

Professional development Program observations Student data (e.g. attendance, grades, test scores, discipline)

The issue is about FIDELITY: - Is the program reaching its target population? - Is the implementation consistent with intended program design? - Is the program on-track? Off-track? - What data-driven adjustments need to be made to the program design?

Ex. 1: Teacher Ratings  Take place during the first week of each month  AS leaders provide assessments on four elements of student progress:    

Treats teachers/staff with respect Acts respectfully towards peers Works diligently on assignments Demonstrates positive attitude toward AS activities

 Monthly assessments are provided for each child  All of the Time, Most of the Time, Some of the Time, Rarely or Never

Ex. 2: Activity Ratings • Take place at various times following recreation activities • Young people provide input on four key areas of development: – – – –

How much FUN did you have in recreation today? (affective) How much did you LEARN in recreation today? (cognitive) How PHYSICALLY ACTIVE were you in recreation today? (physical) How much did you get to play with your FRIENDS in recreation today? (social) – What did you like BEST about recreation today? – What did you like LEAST about recreation today?

Ex. 3: Promising Program Practices  Systematic program observations done at each program site  Use trained observers (Clemson University graduate students)  Observe programs for 15 minutes:  General activity information (# of kids, # of adults, activity type, location)  Open comments: What are youth doing? Materials used? Instructional processes? Overall tone of the session? Extent of engagement? Quality concerns?  Promising Practices Ratings: 1) Supportive relationships with adults; 2) Supportive relationships with peers; 3) Level of engagement; 4) Opportunities for cognitive growth; 5) Appropriate structure; 6) Over-control; 7) Chaos; 8) Mastery orientation Vandell et al. (2005)

Vandell et al. (2005)

Vandell Vandelletetal. al.(2005) (2005)

For Next Week… • Start thinking about Chapter 3 (Methodology). The draft isn’t due for 2 weeks…but think about the things we talked about tonight. Focus specifically on: – Describing and justifying your use of the population that you are interested in studying. – Describing how will you identify your population? Access them? What is the total number of your population? – Discuss your approach to sampling. Will you sample? If so – will you use a single or multistage sampling process? Will you use a population, random, systematic, stratified, or convenience sample? Why? – How will you determine your sample? How will you select and/or recruit participants for your study? – How large will your sample be? Why?

• Research Forum Abstracts are DUE NEXT TUESDAY (4/2) in Box!