Sparking Life Niagara: Program E valuation Report 2011-‐ 2012
“Exercise is the single most powerful tool you have to optimize your brain function.” -‐Dr. John Ratey
Program Evaluation Report
Sparking Life Niagara
2011-‐2012
v TABLE OF CONTENTS
1.0 Introduction and Context 1.1 Background -‐ Dr. John Ratey & SPARK……………………………………………………………………………2 1.2 Program Description -‐ Sparking Life Niagara……………………………………………………………...2-‐3 2.0 Methodology 2.1 Objectives………………………………………………………………………………………………………………………4 2.2 Evaluation Questions……………………………………………………………………………………………………..4 2.3 Methodologies……………………………………………………………………………………………………………5-‐6 i. Outcome Assessment Tools………………………………………………………………………………….………5 ii. Analysis Plan …………………………………………………………………………………………………………….5-‐6 2.4 Limitations……………………………………………………………………………………………………………………..6 3.0 Key Findings 3.1 Success………………………………………………………………………………………………………………………7-‐17 i. Academic Outcomes……………………………………………………………………………………………..…7-‐11 ii. Emotional Well-‐being Outcomes ……………………………………………………………………….………11 iii. Physical Health Outcomes………………………………………………………………………….…………12-‐17 3.2 Design & Delivery ……………………………………………………………………………………………………17-‐20 4.0 Summary of Findings………………………………………………………………………………………………20-‐21 5.0 Key Recommendations 5.1 Evaluation Process…………………………………………………………………………………………………..21-‐22 5.2 Program Implementation………………………………………………………………………………………..22-‐23 6.0 References……………………………………………………………………………………………………………….….24 7.0 Appendices……………………………………………………………………………………………………………..25-‐30 7.1 Measurement Procedures Height………………………………………………………………………………………………………………………….25 Weight…………………………………………………………………………………………………………………………26 Resting Heart Rate……………………………………………………………………………………………………….27 Blood Pressure…………………………………………………………………………………………………………….28 7.2 Norms Blood Pressure Levels for Boys by Age and Height Percentiles……………………………………..29 Blood Pressure Levels for Girls by Age and Height Percentiles……………………………………..30 1
Program Evaluation Report
1.0
Sparking Life Niagara
2011-‐2012
INTRODUCTION
This report presents the findings of the evaluation of the Sparking Life Niagara pilot project for Semester One, from September 2011 to January 2012. Daily activity tracking and student assessments were carried out to measure the effects of the program on mental, emotional and physical health outcomes. This evaluation will summarize significant and relevant findings to determine the extent to which the program is achieving expected outcomes, as well as, identify areas to improve the quality and effectiveness of the program and evaluation process.
1.1 Dr. John Ratey & SPARK
The program is founded on research by Dr. John Ratey, an Associate Clinical Professor of Psychiatry at Harvard University, who has extensively studied and assessed the relationship between physical activity and academic performance in students. In his latest book “Spark: The Revolutionary New Science of Exercise and the Brain,” Dr. Ratey explores the neuroscience behind the potential benefits of aerobic exercise on learning capacity, aging, stress management, and other mental health issues such as anxiety, depression, and attention deficit disorder (ADD). While the advantages of physical activity for general health are often extolled, these findings are unique in that they emphasize exercise as a means to achieve optimal mental performance by sparking brain function to improve learning on three levels: 1) Optimizes mind-‐set to improve alertness, attention, mood and motivation 2) Prepares and encourages nerve cells to bind to one another, storing new information 3) Spurs the development of new nerve cells from stem cells in the hippocampus The first school-‐based program to show how aerobic activity can transform the mind, in addition to the body, was Naperville High School in Illinois. The “Zero Hour” program required students to complete an hour of aerobic activity before first period class. The novel approach to physical education, assessed effort rather than skill, providing students with heart rate monitors so that they could be graded by how much time they spent working out in their target heart rate zone. Students who participated in Zero Hour saw a 17.0% improvement in reading comprehension compared to a 10.7% improvement for students who did not participate in the program (Ratey, 2008). Based on the success of Zero Hour and similar programs, the Sparking Life movement is underway to encourage schools to find novel ways to incorporate aerobic physical activity into the academic setting, as a tool to aid students in learning and increase educational attainment levels. Education is an important social determinant of health and is associated with many measures of population health (PHAC, 2004). A school-‐based physical activity program aimed at boosting educational attainment and laying the foundation for a healthy, active lifestyle during adolescence is a key public health initiative.
1.2 Sparking Life Niagara
Niagara Region Public Health (NRPH) in collaboration with District School Board of Niagara (DSBN), Niagara Catholic District School Board (NCDSB), Niagara Sport Commission (NSC) and Public Health Ontario (PHO) has piloted “Sparking Life Niagara” in two area high schools, Eastdale Secondary School and Lakeshore Catholic High School. The Sparking Life program model incorporates 20 minutes of aerobic activity at the start of the school day, either during Period 1 or 2, and shorter exercise “bursts” 2
Program Evaluation Report
Sparking Life Niagara
2011-‐2012
throughout the day. Students were expected to work out at 65 to 85% of their maximum heart rate and had access to a variety of exercise equipment. The program was initiated this academic year, from September 2011 to June 2012 and Semester One data is now available from the evaluation. Sparking Life Niagara participants earn a Physical Education (PE) credit based on their effort, consistency in terms of completing the daily aerobic activity at their target heart rate (THR) and willingness to participate in assessments. An emphasis is also placed on credit recovery so students can complete work towards other core course requirements. Eastdale Secondary School At Eastdale Secondary School, the Eagle Spark Initiative runs for the first two periods of the school day and is implemented by the English and Drama teacher. Semester One included 12 students aged 14 to 19 years old, 10 males and 2 females. Students selected for the program had identified reading challenges. Forty minutes were allotted for the morning workout, allowing students sufficient time to complete the required 20 minutes of aerobic activity. The aerobic activity was carried out in the school gym where students had access to exercise equipment such as treadmills, elliptical machines and spinning bikes. Each student had a polar heart rate monitor to ensure they were working out within their THR zone. Breakfast was provided to all participants after the morning workout. After morning exercise, literature assignments were handed out and students engaged in various activities including independent reading, oral reading, and performing plays. Two energizers were built into the morning class at logical breakpoints. At lunch time, students were dismissed and continued with their regular course scheduling in various classrooms. Lakeshore Catholic High School At Lakeshore Catholic High School, the Sparking Life Initiative runs in an alternative setting “success”” classroom and is implemented by the Physical Education teacher, who acts as the primary facilitator in charge of running daily workouts and assigning coursework. Administrative support and data collection is carried out by the school Business teacher. There is also a Child and Youth Worker in the class to provide additional support to students. Approximately 80% of the students were considered “full-‐time success students” and remained in the classroom for the whole day, while 20% of students attended class or co-‐op outside of the resource class for a morning or afternoon period. Semester One included 20 students aged 14 to 19 years old, 12 males and 8 females. Students selected for the program were considered “at-‐risk” of not graduating in time for various reasons including behavioural issues and poor academic performance. The aerobic activity was carried out in the equipment room, adjacent to the classroom, where student had access to exercise equipment, as well as, direct access to an outdoor track. Each student had their own polar heart rate monitor that was pre-‐set to their THR zone. Breakfast was provided to all students after the morning workout, as well as, snacks throughout the day. An integral component of the success resource class was credit recovery so formal lessons were not carried out during the day. Students completed online learning courses or worked on course packages to obtain full credit for core courses and recover credits. Teachers were available for support with any of these activities. An afternoon energizer was built into the class when energy levels were declining. 3
Program Evaluation Report
2.0
Sparking Life Niagara
2011-‐2012
METHODOLOGY
2.1 Objectives Ø assess the success and progress of the program Ø assess testing and performance measurements being used Ø assess the evaluation design and program implementation Ø identify key findings Ø make recommendations to improve efficiency and effectiveness of the program and evaluation 2.2 Evaluation Questions Student Outcomes with the Sparking Life Niagara Program § § §
Did Sparking Life Niagara students feel that participating in the program made it easier to learn? Did students notice any changes in their mood after exercise? Did students improve their physical health or interest in physical activity?
Students’ Experience in the Sparking Life Niagara Program § § § §
Did Sparking Life Niagara students feel there was any value or benefit gained from participation? What did students think was the best part about the program? What would students change about the program if they could? Would students want to participate in this type of program again?
Teachers’ Experience in the Sparking Life Niagara Program Did teachers adhere to the Sparking Life Niagara program guidelines? Did the Sparking Life Niagara Team (i.e. program teachers, principals, guidance counsellor, Niagara Sport Commission interns) notice any changes in the students’ behaviour? Parents’ Experience in the Sparking Life Niagara Program § Did parents notice a change in their child’s interest in school? § Did parents notice any changes in their child’s overall mood and behaviour at home? § Did parents notice a change in their child’s interest in physical activity? § Would parents want their child to participate in this type of program again? § §
To answer the evaluation questions above, five research methods were used. These included a student pre/post questionnaire, teacher-‐led discussions with students, a teacher pre/post questionnaire, parent telephone interviews and informal feedback from school principals, guidance counselors and student interns with the Niagara Sport Commission. These methods will be elaborated upon in the Key Findings (Section 3.0). 4
Program Evaluation Report
Sparking Life Niagara
2011-‐2012
2.3 Methodologies i. Outcome Assessment Tools Beep Test The beep test is one of the most commonly used tests to assess endurance by scoring maximal running aerobic fitness. The test, which can be downloaded as a mobile phone application or accessed on Compact Disc (CD), involves continuous running between two lines in time to recorded beeps. Participants stand behind one of the lines facing a second line, 20m away, and begin running when instructed. The speed of each beep starts off quite slow and gains speed with each minute so that at each advancing level, the increment between beeps becomes shorter. The test is stopped when the participant is unable to reach the line for two consecutive ends and the last level achieved is recorded. Kaufman Test of Educational Achievement The Kaufman Test of Educational Achievement, Second Edition (KTEA-‐II) was administered by program teachers to assess students’ numeracy skills. The Comprehensive Form of the KTEA-‐II is an ideal diagnostic tool that can be used as a quick measure of current achievement for individuals aged 4 to 25 years old. It yields norm-‐referenced scores and can be re-‐administered to check progress. The Math Composite is composed of two subsets, Computation and Concepts & Applications. Math Computation assesses basic arithmetic, while Concepts & Applications assesses numerical reasoning skills. Ontario Comprehension Assessment The Ontario Comprehension Assessment (OCA) was administered by program teachers to assess students’ understanding of informational text. The OCA is a non-‐intensive, whole-‐class assessment of reading comprehension for students in Grades 7 to 10. It is an effective tool for measuring progress over time, as it includes two student reading selections for each grade. This allows for one selection to be administered at the start of the grade and the second at the end of the grade. Physical Measurements On-‐site physical measurements were taken by Public Health Nurses at three time points, the start, midpoint and end of the program. Physical measurements were recorded in the metric system and included height, weight, waist circumference, resting heart rate and blood pressure. Weight measurements were taken using a calibrated scale, while height and waist circumference were taken using a measuring tape. Blood pressure and resting heart rate measurements were also recorded using a calibrated heart rate monitor. To ensure accuracy, measurements were repeated multiple times. Detailed measurement procedures can be found in the Appendices (Section 7.0). ii. Analysis Plan Quantitative analyses comprised descriptive statistics and dependent t-‐tests to assess differences between students’ assessment scores at the start and end of the program, at the 5% significance level. Significant differences for categorical academic test scores were assessed at the 5% significance level 5
Program Evaluation Report
Sparking Life Niagara
2011-‐2012
using McNemar’s Test for matched-‐pairs before-‐and-‐after measurements. These were carried out using the Data Analysis Toolpak in Microsoft Excel Version 2010 (Microsoft Corp., Redmond, WA) and corroborated using SPSS Version 16.0 for Windows (SPSS Inc., Chicago, IL). Qualitative analyses, which involved coding and categorizing responses from students, parents and teachers, were performed using Microsoft Excel Version 2010 (Microsoft Corp., Redmond, WA). 2.4 Limitations Low Literacy Skills: Many of the students selected to participate in the program were chosen because they had identified literacy problems. Questionnaire responses and discussion may have been limited by the students’ ability to convey their thoughts and feelings about the program. No Comparison Group: With no available comparison group there is no way to estimate the changes students would have undergone had the program not been implemented. Pre/Post Assessments: With pre/post assessments, it is hard to discern if positive changes are attributed to the program or due to the natural maturation and learning that occurs in students throughout the school year. The purpose of any student assessment, such as the OCA, is to better inform teachers so that future instruction can be tailored to meet students’ needs. Results from the program start helped teachers to recognize gaps and inform future lesson plans so students could perform better in these areas. This would contribute to some of the improvement observed when students were re-‐tested. Program Adherence: Student attendance and lateness on a regular basis would have impacted the amount of daily aerobic activity completed. As well, those students unable to reach 65-‐85% of their THR during exercise may not have experienced the same desired benefits as students that did. Results are not adjusted to accommodate for the varying levels of program adherence among students. Program Opt-‐out: To ensure the voluntariness of participation, students were allowed to opt-‐out of the program or the evaluation process at any point. While efforts were made to determine the reason for opting-‐out, students who remained in the program or responded during the evaluation may represent those who were more successful. As a result, post-‐assessment results may be over-‐inflated. Self-‐report Bias: Questionnaires administered to students, discussions and parent telephone interviews relied on self-‐reports of any changes in participants’ thoughts, feelings, attitudes or behaviours that may be attributed to the program. Participants or parents may not have responded truthfully, either because they could not remember or they wished to present themselves in a socially acceptable manner. Self-‐selection Bias: The pilot project is being conducted in two schools that expressed a desire to participate in the program. The two schools that have chosen to participate may be systematically different from others schools in the region. For example, program teachers may have higher levels of motivation or schools may have access to more resources, such as equipment and support staff. 6
Program Evaluation Report
3.0
Sparking Life Niagara
2011-‐2012
KEY FINDINGS
3.1 Success
i. Academic Outcomes
Percentage (%) At/Above Provincial Std!
² Reading Comprehension At the start of the program, the majority of students (53.6%; n=15) reported that they did not enjoy reading. In addition to a general feeling that it was a boring activity (7), students mentioned that they found reading to be difficult (5). Some students stated that they were slow readers or simply “not good at it.” When students who did enjoy reading were asked why (32.1%; n=9), those who responded discussed the fact that books take you away from reality and “let your imagination grow and spark.” Figure 1. Ontario Comprehension Assessment, Percentage of Students Meeting/Exceeding the Provincial Standard (Level 3 or 4) at Start and End of Program (N = 20) 100 90
Start
80
End
70 60 50 40 30 20 10 0
When reading comprehension was assessed at the start of the program with the OCA test, less than fifty percent of the Sparking Life Niagara students were meeting or exceeding provincial standards on any of the seven components. When the OCA test was administered at the end of the program, the number of students performing at or above provincial standard, defined as a Level 3 or 4, had increased in six of the seven test components. The greatest improvements were seen in students’ ability to classify and organize ideas, as well as, use comprehension strategies when reading. These differences were statistically significant as assessed by the McNemar Test (p
