CHAMPLAIN SCHOOL FACILITATION PILOT PROGRAM 2013-14 FINAL EVALUATION REPORT

SEPTEMBER 2014

ACKNOWLEDGEMENTS The Propel team wishes to acknowledge and thank several people and groups for their contributions to the Champlain School Facilitation Pilot Program evaluation over the last two years. Thanks to: 

Champlain Cardiovascular Disease Prevention Network and the Heart and Stroke Foundation with special thanks to Micheline Turnau, Laurie Dojeiji, Natalie Martin, Greg Killough, Laura Garton and Mark Holland for collaboration with the Propel team to optimize the design and use of evaluation findings.



Intervention and comparison school administration and teachers for participating in the evaluation through school surveys or interviews;



Intervention and comparison school staff who assisted with administration of the online questionnaires to students;



Intervention and comparison school parents and students who completed consent forms and completed surveys;

Suggested citation Yessis, J, Zummach, D, Soni, S, Manske, S, (September 2014). Champlain School Facilitation Pilot Program 2013-2014. Final Evaluation Report. Waterloo, Ontario: Propel Centre for Population Health Impact, University of Waterloo. Prepared by: Jennifer Yessis, PhD, Dana Zummach, BSc, Shilpa Soni, BSc, Steve Manske, EdD *

Propel Centre for Population Health Impact.

For further information regarding this report, please contact: Propel Centre for Population Health Impact Lyle S. Hallman Institute University of Waterloo 200 University Avenue West Waterloo ON Canada N2L 3G1 Telephone: 519-888-4520 Fax: 519-886-6424 Email: [email protected] The Propel Centre for Population Health Impact is a collaborative enterprise that conducts solution‐ oriented research, evaluation and knowledge exchange to accelerate improvements in the health of populations. With more than 34 years of experience in impact-oriented science, our vision is to help transform the health of populations in Canada and around the world. Supported by a Canadian Cancer Society Research Institute (CCSRI) major program grant (2011-2016), the University of Waterloo and more than 30 grants and contracts from federal and provincial governments and NGOs, Propel’s niche is relevant and rigorous science that informs policies and practice to prevent cancer and chronic disease. Propel was founded by the Canadian Cancer Society and University of Waterloo.

Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

2

Table of Contents KEY MESSAGES .......................................................................................................................................... 4 Major accomplishments of Champlain School Facilitation Pilot Program .......................................... 4 Implications for future implementation ................................................................................................ 5 Conclusion .......................................................................................................................................... 5 1. INTRODUCTION ................................................................................................................................ 6 1.1. Purpose and Overview of this Report ...................................................................................... 6 2. OVERVIEW OF CHAMPLAIN SCHOOL FACILITATION PILOT PROGRAM 2013-14 ..................... 6 2.1. Description of program ............................................................................................................. 6 3. OVERVIEW OF EVALUATION .......................................................................................................... 9 4. MAJOR ACCOMPLISHMENTS OF CHAMPLAIN SCHOOL FACILITATION PILOT PROGRAM .. 10 4.1.1. Impact of the pilot at school and student levels ..................................................................... 10 4.1.2. Process learnings for continuous improvement ..................................................................... 13 4.1.3. Conceptual learnings for scaling up ....................................................................................... 15 5. CONSIDERATIONS FOR FUTURE ................................................................................................. 16 5.1. Implications ............................................................................................................................ 16 Conclusion ........................................................................................................................................ 17 APPENDIX Appendix A: Key findings across data collection procedures ........................................................... 18

Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

3

KEY MESSAGES

This report presents evaluation results of the Champlain School Facilitation Pilot Program that was implemented in selected schools in the Champlain region of Eastern Ontario from the spring of 2013 until the end of the 2013/2014 school year. The evaluation was intended to inform continuous improvement of the facilitation pilot, in addition to identifying the program’s outcomes at both school- and student-levels and conceptual learnings that may be used to inform scale-up in Champlain or other jurisdictions. Major accomplishments of Champlain School Facilitation Pilot Program

Impact of the program at school and student levels     



Schools are moving in the right direction, evidenced in their establishment of foundational processes for improving healthy eating and physical activity environments School stakeholders reported student behaviour improved from time 1 to 2 (e.g. increased physical activity, fewer behavioural referrals, increased focus, improved morale) Self-reported student behaviour change was limited; positive examples of changes occurred in schools that focused on sugar-sweetened beverages Student behaviours still have lots of room for improvement Gender was a significant predictor of behaviour related to physical activity and healthy eating. (e.g., boys were more physically active than girls, girls tended to eat healthier foods that boys) Determinants of behaviour (e.g., attitudes, subjective norms) significantly predicted physical activity and healthy eating behaviours

Process learnings for continuous improvement The following critical success factors for advancing healthy school environments emerged:       

Leadership development of leaders within school communities Buy-in and opportunities for involvement from a variety of stakeholders Student voice and leadership Recognizing and addressing competing/conflicting priorities Clear communication Meaningful involvement of community partners (e.g., public health) Facilitator role and personal characteristics, resources/connections provided to schools, and the role taken with each school

Conceptual learnings for scaling up  

Partnerships for projects within the school community setting are essential for implementation and sustainability Evaluation is not only important for accountability and improvement, but also to inspire relevant school-level actions

Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

4

Implications for future implementation



 





Schools involved in facilitation for improving school environments need to commit to a process using a comprehensive school health framework; facilitation must be tailored to the history, capacity and context of each school community. Greater involvement of community partners may support school actions and extend healthy environments outside of the school which is essential for sustainability. Ongoing knowledge exchange opportunities between schools (especially, but not limited to geographically proximal schools) would be valuable in-person or by teleconference. A gathering of participating schools and their respective stakeholders early in the process could facilitate effective sharing. Knowledge exchange may also occur through sharing program information, templates and creative ideas on an online platform. The school change process requires more time than afforded in this pilot. School facilitation and support need to be implemented over a longer time period within each school community. Depending on resource availability for implementation, some schools may begin in one year with greater involvement from the facilitator. As they become more experienced, new schools may begin the following year. Experienced schools can provide support and mentorship to new schools for greater sustainability. While participants may feel evaluation activities are time consuming at the start, their use for school-level action planning (as well as monitoring impact) makes them valuable to most participants.

Conclusion

The Champlain Pilot evaluation made important contributions to school health and our understanding of how to achieve healthy school communities. When considering the pilot’s contributions, schools that were involved firmly established foundational processes necessary for creating healthy school environments. Schools have made positive changes to their school environments in new policies (e.g., daily physical activity, nutrition), new programs (playground leader programs, healthy snack programs, leadership), and school culture. Schools that focused on reducing sugar-sweetened beverages showed statistically significant reductions from time one to time two in consumption of sugar-sweetened beverages on the previous day. The full benefits (sustained change) of the facilitated approach in schools will only be realized with longer-term and wider-spread implementation within the Champlain region and across Canada.

Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

5

1. INTRODUCTION 1.1.

Purpose and Overview of this Report

This report is written for the Heart and Stroke Foundation and presents a summary of the evaluation results of the Champlain School Facilitation Pilot Program that was implemented in schools in March 2013 until the end of the 2013/2014 school year. Baseline evaluation findings were shared in August 2013, and submitted in a baseline report in early 2014. This final report integrates and builds on results from the baseline report. It presents main contributions of the pilot and identifies implications for the future. For additional details on the evaluation, see the Champlain School Facilitation Pilot Program 2013-14 Evaluation Technical Report, September 2014. The evaluation is intended to inform continuous improvement of the facilitation pilot, in addition to the program’s outcomes at both school and individual student levels and conceptual learnings that may be used to inform scale-up. More detailed information on methods and results is available in companion documents which are referenced throughout this report, but especially in the technical report referenced above. 2. OVERVIEW OF CHAMPLAIN SCHOOL FACILITATION PILOT PROGRAM 2013-14 2.1.

Description of program

The Heart and Stroke Foundation (HSF) funded and worked in partnership with the Champlain Cardiovascular Disease Prevention Network (CCPN), and the Propel Centre for Population Health Impact (Propel) at the University of Waterloo piloted an initiative called “Fostering Healthy School Environments: Champlain School Facilitation Pilot Program 2013-14” to support healthy eating and physical activity among school-aged children. Pilot schools, a small cluster of elementary schools (n=16), were provided with seed funding ($1150 for the 2013-14 school year). Schools also received ongoing support (March 2013 to June 2014) from a school facilitator who assisted schools in developing a 2013-14 School Action Plan in at least one of the following priority areas: school nutrition, school travel planning (active transportation) and/or active play. The facilitator provided/linked schools with tools and resources to support successful implementation of the plan. The Champlain Pilot was designed to guide schools in creating environments where healthy behaviours (namely, physical activity, healthy eating, and active transportation) could be easily facilitated, encouraged, supported, and celebrated. At its core, the Champlain Pilot aimed to "kick start" a shift in school culture - one in which health was recognized as a foundation for student success, and over time, would apply a health "lens" across all aspects of school life. As such, the Champlain Pilot focused primarily on modifying physical and social environments, recognizing that there are a number of factors that influence an individual's health behaviours. In many instances, the healthiest choice is not the easiest or most accessible choice; working towards creating supportive school environments helps counteract this. The Champlain Pilot was grounded in a Comprehensive School Health (CSH) approach. In the province of Ontario, CSH is expressed through the Ministry of Education's Foundations for a Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

6

Healthy School framework, recognizing the importance of whole school environments in influencing health behaviours. The framework includes four pillars: 1) High Quality Instruction and Programs, 2) a Healthy Physical Environment, 3) a Supportive Social Environment, and 4) Community Partnerships. Having the Champlain Pilot grounded in this framework meant that all schools worked towards establishing school action plans that were multi-faceted in their strategies and activities, striving to address all four pillars, and tailored to their own school context (resources, history, priorities, etc). Figure 1 illustrates where the facilitator focused most efforts during the Champlain Pilot. The pilot aimed to encourage foundational processes to advance healthy school efforts by modifying the school environment and initiating a shift in school culture to ensure sustained improvements. Examples of foundational processes include having a leader in place, forming a team, working with community partners (such as public health), and working with the whole school community. Figure 1. Focus of the Champlain Pilot

Figure 2 illustrates the steps that the school facilitator encouraged schools to take; the school facilitator worked with each school to encourage these foundational processes, and encourage action and improvements in the school environment. The role of the facilitator, in essence, was to coach schools through a process for planning and implementing health-related goals and activities, which collectively aimed to shift the school environment to be more conducive to the adoption of healthy behaviours. Depending on the school’s stage of readiness, the facilitator also served as a “player coach” when necessary, acting as an extra set of hands in leading schoolwide events and activities. The duration and dose (intensity) of the Champlain Pilot suggested school community improvements would start with foundational processes and changes in school environments where action was taken, and also may influence changes in knowledge, attitudes Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

7

and skills depending on the school’s focus. The team recognized that while behaviour may be influenced by changes to a school environment and education, there is greater likelihood of sustained change when other environments outside the school are also supportive. In addition, changes to individual outcomes like Body Mass Index (BMI) only follow sustained behaviour change, and are influenced by other factors like genetic susceptibility and child development stage. Recognising the time and resource constraints of the Champlain Pilot, determining the impact of the pilot also depended on measuring precursors of the behavioural change we could expect, including attitudes and perceived social norms. Figure 2. Foundational Processes for Advancing Healthy School Communities (“Steps for Success”, CCPN)

Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

8

3. OVERVIEW OF EVALUATION

The primary purposes of evaluation of the Champlain Pilot were to: 1. examine the impact of the program at school- and student-levels; 2. contribute to continuous improvement of the pilot program; and, 3. explore considerations for scaling up and expansion. Evaluation procedures were developed to answer these evaluation questions as documented in Appendix A. The evaluation used a mixed methods approach (both quantitative and qualitative) exploring multiple lines of evidence from different stakeholder perspectives. Pre-(time 1) and post-(time 2) Pilot student-level surveys measured specific eating and physical activity outcomes (attitudes, subjective norms, behavioural intentions and behaviours) as well as body mass index (BMI) from self-reported height and weight. Additionally, to understand experiences in participating in the pilot, as well as factors that supported or challenged school and student-level changes, evaluation staff conducted a series of interviews with principals and program staff over the course of the program, and focus groups with principals and parents. In two selected schools, students engaged in a photovoice project to learn about eating and physical activity environments at their school through students’ eyes. Student questionnaire analyses explored student outcomes at time 1, compared student responses from time 1 to time 2, and examined differences between intervention and comparison group results. Multivariate analyses helped examine multiple predictors of outcome simultaneously. These models used several variables to describe the program. For instance, one variable indicated whether the school had a physical activity focus; another variable indicated whether the school had a nutrition focus (some schools had both types of interventions). Yet another variable reflected the “level” or intensity of engagement in the Champlain Pilot program at the school. Gender and grade were also included in the models. Finally, interactions between the intervention variables and time (time 1 vs time 2) were investigated to see whether there were significant changes over time. Various outcome variables were investigated, including behavioural and behavioural intentions. The findings for each data collection procedure focused on the school environment were summarized individually (see Technical Report for summaries). Summaries were shared with the evaluation team or specific individuals to verify findings; findings were also shared during program implementation to support program learning. After initial sharing, summaries were finalized, and the main themes were examined and synthesized to examine similarities across each procedure. Appendix A contains a table that identifies the main themes for different data collection procedures focused on the school environment. Main themes across data collection procedures have been synthesized for this report into main messages. More detailed results with specific data can be examined in the Technical Report.

Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

9

Evaluation Procedures Informing the Final Report Student-level Quantitative Student survey: Spring 2013 & Spring 2014

     

School Environment Quantitative School environment survey (Healthy School Planner): Spring 2013 & Spring 2014 Qualitative Principal interviews: Spring 2013 & Spring 2014 Principal focus groups: Winter 2014 Parent focus groups (2 schools): Winter 2014 Student photovoice (2 schools): Winter & Spring 2014 CCPN staff and facilitator interviews: Fall 2013, Winter 2014, Spring 2014 Other: Evidence syntheses on best available evidence for facilitated approaches in school settings 4. Major accomplishments of Champlain School Facilitation Pilot Program

Over the course of the 2-year pilot (14 months of work in the schools), findings were grouped by main purpose of evaluation including impact of the pilot at school- and student- levels, process learnings for continuous improvement and conceptual learnings for scaling up. 4.1.1. Impact of the pilot at school and student levels

Schools are moving in the right direction, evidenced in their establishment of foundational processes for improving healthy eating and physical activity environments

Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

10

School-level data were compiled using Healthy School Planner at time 1 and time 2; feedback from qualitative data collected also informed this main finding. Based on the actions schools took from time 1 to time 2, schools involved in the pilot have firmly established foundational processes required for a comprehensive approach to improving healthy eating and physical activity environment. For example, all pilot schools had developed an action plan related to their priority(s) of focus (active play, healthy eating, active transport), all pilot schools had a leader to advance their work, and more pilot schools had established plans for succession in case the leader left the school. In developing their action plans, pilot schools used their data to inform their plans. Not only did schools have an action plan, but they took actions to advance their efforts to change the school environments. Pilot schools had more meaningful partnerships with organizations such as public health and other community partners (e.g., Green Communities Canada). These organizations provided support for school action. Pilot schools had greater awareness and engagement among staff, students and parents about working together to create a healthy school environment. One area for improvement identified at baseline (time 1), was in engaging students in meaningful ways including providing leadership opportunities. Subsequently, Pilot schools were documented in providing more leadership opportunities to students in programs like Playground Activity Leaders in Schools (PALS) and in planning events such as a health and wellness fair.

Two Schools Water Down SugarSweetened Beverage Consumption Two schools focused on reducing consumption of sugar-sweetened beverages (SSBs) at their school. One of the schools created a policy allowing only water in classrooms, educated the school community (e.g., in agenda, newsletter, and from principal in announcements,) and discouraged consumption of pop, sports drinks and other SSBs. The proportion of student SSB consumption was reduced from 67% to 58% over the course of the pilot at this school. At an intermediate school, one of the school goals was to decrease the consumption of high sugar, high fat foods and drinks. Students were not permitted to sign out of class (or school) to limit the purchasing of snacks and drinks from a local convenience store. As well, water was the only drink allowed in class. To educate students, a presentation entitled “sugar shocker” was made to all grade 7 and 8 students describing the problems with sugar in drinks, and the challenges of artificial sugar, real sugar and energy drinks. The proportion of student SSB consumption showed a trend of improvement from 67% to 63% drinking SSBs on the previous day. Students also reported consuming fewer baked sweets, frozen desserts, and candy or chocolate on the previous day.

Pilot schools also reported advancing efforts to change their school environments. For example, schools reported new food/beverage policies or greater attention to implementing the policies as they were intended. More pilot schools implemented mandated daily physical activity and schools increased minutes of physical education compared to baseline. Pilot schools also provided greater access to healthy food

Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

11

through school breakfast programs, healthy snack programs and events where healthy food was promoted. Student behaviour change from time 1 to 2 reported by stakeholders Stakeholders (i.e., principals, parents) reported observing more physical activity at schools by students and they attributed this to the pilot. There were reports of fewer behavioural referrals as a result of the pilot; one school tracked behavioural referrals before and after the pilot and noted a reduction in them as a result of the program. Other schools reported that students were busy and engaged and exhibited greater focus and these stakeholders attributed these factors to the pilot program. Finally, in other schools the culture and morale of students and staff reportedly improved; the pilot was believed to have moved the school community to work towards a common vision with purpose that resulted in a sense of excitement. Self-reported student behaviour change from time 1 to 2 was limited; positive examples of changes in schools that focused on sugar-sweetened beverages Overall, there were very few statistically significant changes from time 1 to time 2 in behaviours. Results did point to a strong and consistent reduction in consumption of sugar-sweetened beverages from time 1 to time 2 with a stronger impact seen in pilot schools than in comparison schools. In pilot schools, 63% of students reported consuming sugar-sweetened beverages on previous day, compared to 76% in comparison schools. There is still lots of room for improvement in student behaviours As first reported in the baseline survey, student healthy eating and physical activity behaviours continue to be far short of national guidelines at time 2. Our data indicates extremely high percentages of students eating junk food (93% pilot, 95% comparison), a smaller proportion consuming 3 servings of milk a day (30% pilot, 33% comparison) than ideal, large percentages drinking sugar-sweetened beverages (63% pilot, 76% comparison), small percentages meeting physical activity guidelines (60 minutes of moderate to vigorous physical activity per day on 5 or 7 days per week) (32% pilot and 27% comparison) and about half reporting less than 2 hours of screen time (47% pilot and 51% comparison).

Gender was a significant predictor of behaviour related to physical activity and healthy eating Time 2 results showed strongly statistically significant gender effects on physical activity and healthy eating behaviours that were also observed at time 1. These findings are similar to what the baseline report presented. For example, males were more likely than females to meet physical activity guidelines. Females were more likely than males to achieve no more than 2 hours of screen time a day. Females were also more likely than males to eat breakfast, and to eat fruit and vegetables six times per day. On the other hand, males were more likely than females to consume 3 servings of milk each day.

Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

12

Determinants of behaviour (e.g., attitudes, beliefs, subjective norms) significantly predicted physical activity and healthy eating behaviours Multivariate models showed that determinants of behaviour such as attitudes, beliefs or subjective norms significantly predicted physical activity and healthy eating behaviours. For example, confidence in choosing healthy foods at home and school, as well as feeling students could “eat healthy foods if I wanted to”, were consistently significant predictors of eating fruits and vegetables six or more times daily. Those with higher confidence were more likely to achieve this benchmark. The same variables also predicted students who achieved at least 60 minutes of physical activity on 5 days per week. All were positively associated with the behaviour, so those who strongly liked active play, who have a lot of friends that play actively, and who believed they could play actively if they wanted to were more likely to achieve the physical activity benchmark. 4.1.2. Process learnings for continuous improvement

Critical success factors for advancing healthy school environments emerged across data collection procedures and stakeholders in the pilot. As described, the focus of the Champlain Pilot was primarily on creating changes to the foundational processes needed to create a healthy school environment. Evaluation findings were synthesized and the following important factors emerged as themes in assisting schools in using a comprehensive approach to advance physical activity and healthy eating environments. While these factors may not “cause” student attitudes, beliefs, norms, or behaviours, they do lay the groundwork for supporting those types of changes. We identified the following success factors from the data collection procedures of the Champlain Pilot. They should also be considered as important strategies for future implementation. The success factors are listed in bold font below; the themes are described based on understanding of synthesized findings across data collection procedures. Leadership development within school communities Staff and volunteer turnover and burn-out are a reality in schools and create challenges for sustainability. By building leadership capacity within a school community, schools were better able to sustain the process of creating healthy environments. It is important for schools to have an identifiable leader, but should not depend solely on one leader. A formalized committee was not essential for success; however schools that involve a variety of stakeholders showed greater traction within the school community and are likely to have longer term success. Creating leadership succession plans and involving multiple leaders in different ways can also be important factors in sustaining school-level actions. Buy-in and opportunities for involvement from a variety of stakeholders Buy-in from a variety of stakeholders (i.e., staff, parents, students, community group) helped schools support healthy school activities. Buy-in and support from school administrators was essential, but school administrators should not be the sole champion at the school. By engaging diverse stakeholders throughout the process, schools have greater support in planning, organizing and running events that appeal to a broad population at the school. Pilot schools used a variety of strategies to create opportunities for stakeholder involvement (e.g., polling or surveying students to determine what times of activities they would be interested in participating in, varying the time of day or day of week to encourage new groups of parents to be involved). Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

13

Student voice and leadership Engagement of students throughout the planning, implementation and participation process allows schools to better support activities that interest students and expand the reach to include a wider proportion of the student population. Creating opportunities for students to develop leadership skills builds student interest and ownership of initiatives. Schools involved students in a variety of ways including planning initiatives like a health fair, in preparing food for a school-wide dinner in the evening, in planning meals for a school lunch program, and in mentoring and coaching other children in active play. Addressing competing/conflicting priorities The school environment, schedule and people within the school community are very busy. Schools need to find the right balance between all of the demands placed on them and find ways to prioritize creating a healthy school environment (e.g., share leadership, taking a more comprehensive approach to creating a healthy school rather than implementing numerous new activities). Similarly, schools sometimes have priorities that conflict with the goal of creating a healthy school environment. For example, fundraising is needed to support school activities, but schools sometimes rely on selling unhealthy food (e.g., chips, pop, candy) because they sell well. Taking a more comprehensive approach and ensuring that all stakeholders, like parent and student council, are aware and engaged in school health priorities is necessary to ensure that there is a culture of health at the school (e.g., it becomes the way of doing business). Clear communication Because the school environment and stakeholders are so busy, concise and regular communication is necessary for creating awareness, developing buy-in and supporting engagement in creating a healthy school environment. To reach different audiences, communication may take different forms such as a newsletter article, social media, outdoor signage, school announcements, letters home to parents, etc.

“Positive promotion also helps. If the people [school champions] who are promoting it are excited and engaged that helps with the whole process.” Principal

Meaningful involvement of community partners Community partners, while sometimes a challenge for schools to engage, can provide valuable resources for schools (e.g., people, resources, infrastructure, knowledge, funding). Over the course of the pilot, schools began to think of partners as more than just a source of funding, but in terms of what else they may be able to provide such as tool kits, public health expertise.

Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

14

Facilitator is essential and needs strategic consideration in terms of personal characteristics, resources /connections provided to schools, and the role taken with each school The personal characteristics of a facilitator can influence his/her ability to engage and work with schools. Schools tend to rely on the facilitator, especially when the facilitation process is first introduced. The facilitator provides feedback and input when “For a facilitator to bring [ideas] to developing action plans and links schools to community you, that’s a big, big time saver and resources. The facilitator also provides schools with a sense of more likely to get the benefits and accountability to keep working towards their goals. For many the successes that we’re looking schools, the facilitator’s check-in meetings and reminders for.” – Connie Dick, Principal, St. provided support and accountability needed to make sure that Thomas the Apostle implementation of their action plan was on track. 4.1.3. Conceptual learnings for scaling up

Partnerships for projects within the school setting are essential Underpinning the Champlain Pilot is a context that began with a strong set of partners. The Champlain Cardiovascular Disease Prevention Network is an alliance of health and community partners from across the Champlain region representing public health, specialty care, primary care, hospitals, education, community, industry, and academia all working together to reduce the burden of cardiovascular disease in the region. In 2009, the nine Directors of Education and four Medical Officers of Health in the Champlain region signed “The Champlain Declaration: A Call to Action for Physically Active and Healthy Eating Environments in Schools”, formalizing the commitment to work together to establish healthy school environments. Since signing the Declaration, CCPN has been working collaboratively to support healthy school environments in different ways, including this pilot. Throughout this pilot, the established regional partnerships have been influential in ensuring its success – enabling us to work with schools in the Champlain region with support of the school boards and public health as partners in the school communities. Additional partnerships have been instrumental in ensuring the work of the pilot is beneficial including the three-way unique relationship between the CCPN, HSF and Propel in funding, designing, implementing and evaluating the pilot. Evaluation is not only important for accountability and improvement, but also to inspire relevant school-level actions Over the course of the pilot, evaluation results were used for “We highly valued student level continuous improvement and to understand impact. Summaries data assessing attitudes and of evaluation results were shared shortly after data were anecdotal evidence, as well as collected with pilot partners for learning purposes. Baseline data school climate [HSP] data. We want served to inform the encouragement of schools by the facilitator to reflect the "whole child", and not on ensuring foundational assets such as leadership, action plans, just focus on academics. The Propel engagement of students and community partners were in place to data have provided, in a way, a first support their plans. step in helping us collect data outside of academics.” Principal

Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

15

An important conceptual lesson from this pilot is that evaluation results that were fed back to schools in the form of a specific student-level report called a School Profile (i.e., summary of student results from each school) and a school-level report rubric from Healthy School Planner. Both of these reports were very useful to both the facilitator and the school in informing the action plans of each school. Schools also plan to use the time 2 results in continuing their school actions.

5. CONSIDERATIONS FOR FUTURE

Implications of the evaluation findings are organized into next steps for Champlain Pilot Program implementation and evaluation priorities. 5.1.











Implications

Schools involved in facilitation for improving school environments need to commit to a process using a comprehensive school health framework; facilitation continues to be tailored to history, capacity and the context of each school community. Greater involvement of community partners may support school actions in schools, and extend healthy environments outside of the school which is “Having the program run for at least essential for sustainability. 2 years would be helpful. The first Ongoing knowledge exchange opportunities between schools year we are just getting everything (especially, but not limited to geographically proximal set up, learning, organizing and schools) would be valuable in-person or by teleconference. A getting people involved. You feel like gathering of participating schools and their respective you are just getting there by the end of the year. If you can't find the stakeholders early in the process could facilitate effective financial support after the first year sharing. Knowledge exchange may also occur through sharing program information, templates and creative ideas on an online some people may just let it fizzle. If the program ran for 2 years it would platform. School facilitation and support needs to be implemented over a provide the financial support needed for both years and that longer time period (more than two years) within each school would help people maintain it and community because culture change and changes to schoolbe in a better position to continue it level environments take time. Depending on resource on their own after that. In the availability for implementation, some schools may begin in second year I think you would be one year with greater involvement from the facilitator, and more knowledgeable and other schools may begin the following year. Experienced committed to keeping it going.” schools can provide support and mentorship to new schools for Principal greater sustainability. While participants may feel evaluation activities are time consuming at the start, their use for school-level action planning (as well as monitoring impact) makes them valuable to most participants.

Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

16

Conclusion

Results of this evaluation suggest important contributions from the Champlain Pilot. When considering the pilot’s contributions, schools that were involved firmly established foundational processes necessary for creating healthy school environments. Schools have made positive changes to their school environments in new policies (e.g., daily physical activity, nutrition), new programs (playground leader programs, healthy snack programs, leadership), and school culture change. Schools that focused on reducing sugar-sweetened beverages showed statistically significant reductions from time one to time two in consumption of sugar-sweetened beverages on the previous day. The full benefits (sustained change) of the facilitated approach in schools will only be realized with longer-term and wider-spread implementation within the Champlain region and across Canada.

Propel Centre for Population Health Impact: Final Evaluation Report on Champlain Pilot

17

APPENDIX A: METHODS Data collection procedures were used to gather information from school staff, students, parents and implementation staff to obtain a more accurate understanding of implementation and outcomes of the pilot from different perspectives. The findings will be framed in the final report around the main purposes of the evaluation. School Recruitment and Selection

Schools from 5 school boards in the Champlain region were invited to apply to participate in the Fostering Healthy School Environments: Champlain School Facilitation Pilot 2013-14. Additionally, one private school was also invited to apply to participate. Three exclusion criteria were applied before identifying schools for either the intervention or comparison conditions:   

school did not contain at least one of grades 5, 7 and 8 location: the school was ‘geographically isolated’ and would put too much strain on travel time and costs for facilitator to reach the school school size: school population was less than 20 students in any grade 5, 7 or 8 classes

The pilot team from the Champlain Cardiovascular Disease Prevention Network (CCPN), HSF and Propel selected 19 schools in February, 2013 after applications had been reviewed for eligibility. The pilot team designated 16 intervention schools and 3 comparison schools. Schools were selected for the pilot based on their application to participate in the program, school size, school grades (and number of students within the grades we are evaluating), geographic representation and area of focus for the pilot. Schools that applied to participate in the pilot but were not selected as intervention schools were invited to participate as comparison schools. Three schools agreed to participate as comparison schools in the first year. The pilot team reached out to schools through school board partners for additional comparison schools, and 3 more schools signed in the fall of 2014 (for a total of six comparison schools). Data Collection

The evaluation used a mixed methods approach that examined school readiness and pilot expectations, facilitators and barriers to success through interviews and focus groups, and impacts through student and school level surveys to gather an understanding of how the pilot was implemented within the school settings, and to examine school- and student-level changes over time periods.

Propel’s Technical Report on Champlain Pilot

18

Table 1: Data collection methods for evaluating the pilot Methods

Topics

Time

Principal interviews

School readiness for pilot Pilot expectations Pilot experience Recommendations for program improvement

Spring 2013 & Spring 2014

Student survey (grades 5, 7, 8) Self report height and weight, attitudes, social norms, behaviours related to physical activity, healthy eating, positive mental health

Spring 2013 & Spring 2014

School environment survey (Healthy School Planner)

Foundations of healthy school community, healthy eating, physical activity environments and opportunities at the school

Spring 2013 & Spring 2014

Principal focus groups

Feasibility and implementation of program

Jan to March 2014

Parent focus groups

Parent perspectives of the pilot, and of healthy eating and physical activity environment at the school and related school activities

Feb to March 2014

Student photovoice

Student perspectives about the school environment

Winter 2014 & Spring 2014

CCPN staff and facilitator interview

Pilot implementation

Fall 2013, Winter 2014, Spring 2014

School-level data collection

The Champlain Pilot was grounded in a comprehensive school health (CSH) approach. In the province of Ontario, CSH is expressed through the Ministry of Education’s Foundations for a Healthy School framework, recognizing the importance of four pillars and important principles in influencing health behaviours. Data were collected at the school level to assess the school environment, the physical or social environment, community partnerships and teaching/learning within or outside the classroom setting. These topic areas are consistent with Ontario Ministry of Education's Foundations for a Healthy School framework. Principals and key contacts within the

Propel’s Technical Report on Champlain Pilot

19

schools also provided insight into school level changes through an online tool, focus groups and interviews. Healthy School Planner

Healthy School Planner (HSP) is a free, online tool co-developed by the Pan-Canadian Joint Consortium for School Health and Propel to assess the capacity and performance of schools to support healthy environments. The principal (or school contact) was encouraged to invite other members of the school community to assist with the completion of this survey. We asked school teams to complete three modules of the Healthy School Planner: Foundational, Physical Activity Express, and Healthy Eating Express (see Appendix A-C for copies of the questionnaires). The Foundational Module asks questions about the infrastructure and processes in place at the school in order to develop a healthy school community (e.g., Does your school community have a leader for healthy school initiative?). The Foundational Module includes 21 indicators for reporting. The Physical Activity Express and Healthy Eating Express Modules ask questions related to these specific topic areas focused around teaching and learning, the social and physical environments, healthy school policy, and community partnerships. The Physical Activity Module includes 6 indicators and the Healthy Eating module includes 8 indicators. Upon completing each module of the Healthy School Planner, the school received a report scoring each indicator. The scores are presented as a rubric, a format familiar to schools since rubrics are used in assessing student achievement. Level 1 in the rubric represents the lowest level, while level 4 is the highest. Schools completed HSP in the spring of 2013 and again in the spring of 2014. Table 2 shows the completion rates for both rounds of data collection. Table 2: Completion Rates for the Healthy School Planner Foundational Module Time 1

Intervention Comparison

16 schools (100%) 4 schools (67%)

Physical Activity Express Module 15 schools (94%) 3 schools (50%)

Time 2

Intervention Comparison

14 schools (88%) 5 schools (83%)

14 schools (88%) 4 schools (67%)

Healthy Eating Express Module 15 schools (94%) 3 schools (50%) 12 schools (75%) 4 schools (67%)

Principal interviews

Principals participated in two interviews about the Champlain Pilot: beginning (Spring 2013) and end (Spring 2014). All intervention school principals (or designated school contact) were invited to participate in a 30 minute telephone interview at the beginning of the program. The interviewer followed a semi-

Propel’s Technical Report on Champlain Pilot

20

structured interview guide that included questions related to the application process, the pilot focus selected by the school (healthy eating, active play and/or school travel planning), and plans to ensure the success of the program at the school. Baseline interviews were completed between April 4, 2013 and July 2, 2013. See Appendix G for the baseline interview guide. Follow up interviews included questions to understand principals’ and key contacts’ experiences with the Champlain School Facilitation Pilot program, recommended changes for improvement, and factors that may help with sustainability of the program in the school. Follow up interviews were completed with all school principals (or designated school contact) between June 2, 2014 and July 10, 2014. See Appendix H for the follow up interview guide. Each interview was audio recorded. The audio files were used to create detailed notes by question for each interview. The notes were then reviewed and coded independently by two analysts. Once the transcripts had been independently coded, the two analysts and the investigator reviewed codes and deliberated where there was disagreement. The codes were then compiled into the main points that emerged across interviews in each section of the moderator’s guide. Principal focus groups

Principals and key contacts were invited to participate in a focus group as part of the evaluation of the pilot. They selected from 2 dates based on their availability. Twelve of 16 school principals or key contacts participated in these groups. Three principals were not available on either date and one principal did not respond. The three focus groups were conducted via teleconference and audio recorded between February 12 and 27th, 2014. See Appendix I for the moderator’s guide for the principal focus groups. The audio files were transcribed. Each transcript was then coded independently by two analysts. Once the transcripts had been independently coded, the two analysts reviewed codes and deliberated where there was disagreement. The codes were then compiled into key themes. Student-level data collection

Data were collected at the student level at the beginning and end of the program to assess changes in attitudes, beliefs and behaviours related to physical activity and healthy eating. Student survey

Since 2000, School Health Action Planning and Evaluation System (SHAPES) modules have been administered in over 2,500 Canadian schools, with over 430,000 students. These modules have been used to collect and assess information on student behaviours and attitudes in the areas of tobacco use, physical activity, healthy eating and mental fitness. These data are then used to generate profiles to help schools, public health and communities take action to improve the health of young people. SHAPES was developed by scientists at the Propel Centre for Population Health Impact, in partnership with a community of researchers across the country. SHAPES makes it practical to collect standardized youth health data at the school, community, provincial or national levels, and at all these levels simultaneously. SHAPES can provide reports back to each of these levels (assuming sufficient sample sizes) to support planning, evaluation, interventions and research. It Propel’s Technical Report on Champlain Pilot

21

also makes it possible to examine the interacting effects of interventions at various jurisdictional levels. Students in grades 5, 7 and/or 81 (as appropriate for the school population) were asked to complete an online SHAPES survey that included questions about attitudes, and behaviours related to physical activity (including active play and active transportation), healthy eating and positive mental health, and student height and weight. All participating students did so with parental permission. Baseline student data collection was completed between April 16, 2013 and June 25, 20132. Students in grades 5, 7 and/or 8 in the 2014 school year were asked to complete the survey at the end of the program. Student data collection was completed between April 22, 2014 and June 13, 2014. A copy the student survey is provided in Appendix J. Student participation rates for intervention and comparison schools are shown in Table 3. Intervention and comparison schools had similar rates of participation. While it would have been desirable to have more children participate in the survey, these rates of completion (~50%) are similar to other surveys we have conducted using the same method (i.e., active consent permissions forms signed by parents and returned to school to enable student participation). Student self-report may exhibit some bias (e.g., social desirability and simply not knowing may result in underestimates of weight, overestimate of height) compared to observational (regarding behaviours) or objective measures (measuring height and weight directly); however, a similar study comparing the same self-report questions to objective measures in grade 9 students were found to have reliable estimates of height, weight and dietary intake3. Compared to ninth grade students, our participating grades 5, 7 and 8 students are younger and may provide less reliable self-reports. It was recommended that parents check participating students’ height and weight the night before completing the student survey to improve reliability of self-reported height and weight in this age group.

1

To reduce school burden, sixth grade students were not surveyed since they participate in annual EQAO testing. Four schools completed data collection during the Fall 2013 as comparison schools. These data were combined with other comparison schools and used as part of the aggregate baseline data in the Final Report. 3 Leatherdale, S.T., & Laxer, R.E. (2013). Reliability and validity of the weight status and dietary intake measures in the COMPASS questionnaire: are the self-reported measures of body mass index (BMI) and Canada's food guide servings robust? International Journal of Behavioral Nutrition and Physical Activity, 10(1), 42. 2

Propel’s Technical Report on Champlain Pilot

22

Table 3: Completion rates for student survey, by intervention and comparison schools Total Student Population in Eligible Grades

Permission Completed Forms Questionnaires (Yes / Total Received)

% of Total Eligible Population with Data

% of Total Population with Permission with Data

Time 1

Intervention

1495

828/914

763

51%

92%

Comparison

194

129/138

125

64%

97%

Time 2

Intervention

1359

792/852

776

57%

98%

Comparison

194

110/117

109

56%

99%

Student photovoice Introduction

The photovoice component had two purposes: 1. to understand students’ perceptions of healthy eating and/or physical activity influences in the school; and, 2. to identify student recommendations to improve the healthy school environment. The findings from the photovoice discussions will feed into the broader focus of the evaluation, which is to guide ongoing refinements and improvements to the program, to measure the impact in schools, and to determine how the existing design of the program may be scaled up/expanded for broader implementation across jurisdictions in the future. Methods

Photovoice is a participatory method that involves participants in identifying and mapping their needs. Participants engage in facilitated discussions to reflect on pictures they have taken of their environment and create action plans to promote healthy behaviours. The information is shared with policy makers to be used for future planning. This participation develops personal meaning of the data and serves as evaluation data for the program. Photovoice was conducted with one grade 5 and one 5/6 split class at two separate schools. The two schools were purposely selected from a pool of schools with a focus on engaging students. This method served evaluation and intervention purposes. In selecting students to participate in photovoice, principals were contacted, followed by a classroom teacher at the school who agreed to participate in the project. Information letters and consent forms were shared with students and parents of students involved in each of the photovoice classes. The first discussions took place at each school on March 25th and March 26th, with the school facilitator. During this session, students were oriented to photovoice, the cameras, and safe and appropriate behaviours for taking pictures in a school environment. This included asking others for permission before taking their picture, understanding appropriate locations to take the camera, and not downloading pictures to personal computers. The facilitator also led an initial discussion surrounding the four pillars of Comprehensive School Health (CSH) programs (social environment, physical environment, community partnerships, teaching and learning). Students were encouraged to use the cameras to capture factors that influence their physical activity and healthy eating within the school, and fit them into one of the four categories before the next discussion. Students were paired up and given one camera per pair for three days. They were asked to select eight different pictures, which fit Propel’s Technical Report on Champlain Pilot

23

into at least three of the four CSH categories, and submit to the facilitator for the next discussion. Students were also asked to submit an information sheet describing their eight pictures and the CSH pillar each picture belonged to. A detailed facilitator guide can be found in Appendix L. The purpose of the second session was to discuss photos selected by each student pair. The facilitator encouraged each pair to explain why they chose a particular picture, how each selected image influenced their school, and how they might group each image according to the pillars of comprehensive school health. Once the photos were described, the facilitator asked the students for insight into changes they might suggest to make their school a healthier environment. The facilitator planned to take students’ recommendations to principals to spark discussions for change. The third and fourth sessions were planned one month later before the end of the school year. The purpose of the third session was to reorient students to the purpose of the photovoice project, and how to use the cameras. Comprehensive school health was described to students again including descriptions of each of the pillars of the approach. Cameras were provided to the same pairs of students for three days. After three days with the cameras, they were asked to submit eight photographs that they felt fit into three of the four CSH pillars, with an information sheet. During the fourth and final session students had the opportunity to discuss why they took each picture, how the pictures affected their physical activity/healthy eating, and what CSH pillar each picture was aligned with. Students also had the opportunity to suggest changes to promote a healthier school environment at their school. Photovoice Timeline:

Session 1: Orientation

Session 2: Discussion

• What is photovoice? •How to use cameras •Description of CSH pillars

•Why did you take each picture? •What CSH pillar(s) does it belong to? •How would you change things to positvely effect your PA/HE? •Week 2

•Week 1

Session 3: Orientation Recap •How to use cameras •CSH Pillars •Recap main points from session 2

•Week 7

Session 4: Discussion •Why did you take each picture? •What CSH pillar(s) does it belong to? •How would you change things to positively effect your PA/HE? •Week 8

To wrap up the project, the facilitator asked the students and teachers to reflect on what they have learned during the photovoice sessions. The facilitator also planned to take all recommendations and feedback forward to principals and/or key contacts. Observers were present during discussions two and four and took detailed notes to capture the students’ thoughts. The notes were then reviewed and coded by an analyst. The codes were then compiled into the main points that emerged.

Propel’s Technical Report on Champlain Pilot

24

Parent-level data collection Parent focus groups

Two focus groups were conducted with parents from intervention schools involved in the program. We purposively sampled two schools that were actively engaged in the program based on their planned activities, level of engagement with the facilitator and stakeholders involved, according to the facilitator. Over a 2-week period, parents were recruited through multiple methods and with assistance from principals. Information letters and recruitment posters were sent home with students, and shared via email, Facebook and through networks such as School Council/parent committees. An incentive was provided to encourage parents’ participation ($25 gift card honorarium from a local grocery store). As well, child care was offered to parents at the recruitment stage if needed. Refreshments were served at each focus group. Target for each school was 1012 participants. Two separate focus groups were held - one at each school. Nine and 5 parents participated in the focus groups at School 1 and School 2 respectively. Participants were primarily female, highly educated, and primarily working as displayed in Table 4. A scientist moderated the first focus group and a research assistant took notes. The second focus group was moderated by the same research assistant, with a volunteer present to take notes. Both sessions were audio recorded. The audio files were transcribed. Two trained analysts then coded each transcript independently. Once the transcripts had been independently coded, the two analysts reviewed codes and deliberated where there was disagreement. The codes were then compiled into key themes that are described in the findings below.CCPN Intervention staff data collection Interviews with the CCPN staff member and school facilitator provided context and better understanding about implementation of the pilot. The interviews focused on the feasibility and implementation of the initiative to create healthier school environments. They were meant to track learnings from implementation in a systematic way throughout the duration of the pilot. Interviews with CCPN manager

Telephone interviews were conducted with the CCPN staff member responsible for overseeing the implementation of the Champlain School Facilitation Pilot Program. Three interviews were completed during the implementation period: September 2013, January 2014 and June 2014.The interviews were conducted by the Propel investigator; the Propel project manager took field notes during each interview. The field notes were used to examine the main points by question for each interview. Interview recordings were reviewed to supplement the field notes as needed. The notes were then compiled into the main points that emerged across interviews in each section of the moderator’s guide. Interviews with school facilitator

Telephone interviews were conducted with the school facilitator over the course of the implementation period: September 2013, January 2014 and July 2014.The interviews were conducted by the Propel investigator; the Propel project manager took field notes during each interview. The field notes were used to examine the main points by question for each interview. Interview recordings were reviewed to supplement the field notes as needed. The notes were then compiled into the main points that emerged across interviews in each section of the moderator’s guide.

Propel’s Technical Report on Champlain Pilot

25

Data Analysis

Data analysis included both quantitative and qualitative data analysis. Quantitative data analysis School-level environment data – Data from all schools were aggregated to examine the frequencies for each question and indicator at each time period, and grouped by intervention or comparison group. Student-level data (surveys) - Data from all schools were aggregated to examine the frequencies of each question at each time period, and grouped by intervention or comparison group. The frequencies were examined to better understand the range of responses and the questions that had missing information. When questions were not answered by more than 10% of the sample, they were examined more closely to better understand why. Height and weight questions were the only questions with more than 10% of responses missing. Grade 5 classes had higher percentages of missing data. When classes were reminded to measure their height and weight at home, there were fewer missing responses to these items. Selected questions were grouped by indicators and analyzed. Cross-tabulations were conducted between intervention and comparison group, between time 1 and 2, school board, grade, gender and pilot focus (active play, active transport, school nutrition). Frequencies of the various variables were run, and each variable could be cross-tabulated with time (1 vs. 2) and a chi-square test performed to investigate whether there were any differences between time 1 and time 2 data. For some variables, numbers were too small to produce a reliable chi-square statistic. Multivariate analyses were conducted so that multiple predictors could be looked at simultaneously. There were several variables used to describe the intervention – a variable to indicate whether the school had a physical activity intervention, another variable to indicate whether the school had a nutrition intervention (some schools had both types of interventions), and a variable to indicate the implementation level of the intervention at the school (coded as 1, 2 or 3 by the facilitator) based on criteria identified by partners (i.e., focused goal, use of survey data for plans, evolution of plan throughout year, completion of activities, plans for sustainability, comprehensiveness of plan). (All these variables would be coded as 0 for a comparison school). Gender and grade were also included in the models. Finally, interactions between the intervention variables and time (time 1 vs time 2) were investigated to see whether there were significant changes over time. Various outcome variables were investigated, including behavioural outcomes, as well as behavioural intentions. Multivariate models incorporate methods to account for the clustered data by school. Interventions were delivered at the school level. Also, we might expect students at the same school to be more similar to each other in various characteristics than students at other schools (school environment, socio-demographic characteristics, etc.). Therefore, models included a correction technique to account for this clustering. Qualitative data analysis – Principal interviews, principal focus groups, CCPN staff interviews and facilitator interviews were examined for main ideas and themes for each question asked. Once quantitative and qualitative data were analysed, the findings were framed according to the main evaluation purposes.

Propel’s Technical Report on Champlain Pilot

26

APPENDIX B: KEY FINDINGS ACROSS DATA COLLECTION PROCEDURES Finding Moving in the right direction / changes in schools

Principal Principal FG Interview   Critical Success Factors

  Shared responsibility/create leadership capacity (e.g., team, leader)   Buy-in and involvement from a variety stakeholders   Creating opportunities for involvement throughout program leads to greater program reach   Planning for sustainability   Clear communication needed   Involvement of community partnerships is essential   Importance of student voice / leadership opportunities  Address conflicting/competing priorities (e.g., fundraising needs vs. healthy foods, many demands in schools)  A range of involvement from stakeholders (e.g., participation vs. planning/coordinating/organization) Facilitator Specific Considerations   Personal characteristics of facilitator   Role facilitator played in school varied across schools   Connections and resources provided by facilitator to schools  Schools feel accountable to facilitator  Schools needed facilitator support for action planning   Changing level of engagement of facilitator over time due to capacity developed within schools Consideration for Future Implementation Culture change takes time Knowledge exchange opportunities between schools Program needs to run for a longer time period (e.g., 2 to 3 years or longer) Evaluation activities time consuming, but valuable for action planning Propel’s Technical Report on Champlain Pilot

 

CCPN Staff Interviews 

    

Facilitator Interviews

       



    

 27



  

 

 





HSP 

  

Parent FG



Propel Centre for Population Health Impact Lyle S. Hallman Institute, University of Waterloo 200 University Ave. W., Waterloo, Ontario, Canada N2L 3G1 Telephone: (519) 888-4520 • Fax: (519) 746-8171 E-mail: [email protected] • www.propel.uwaterloo.ca

The Propel Centre for Population Health Impact was founded by the Canadian Cancer Society and the University of Waterloo.