Does Teacher Professional Development Have Effects on Teaching and Learning? Evaluation Findings from Programs in 14 states

Does Teacher Professional Development Have Effects on Teaching and Learning? Evaluation Findings from Programs in 14 states 2008 The Council of Chi...
Author: Abigayle Scott
1 downloads 1 Views 644KB Size
Does Teacher Professional Development Have Effects on Teaching and Learning?

Evaluation Findings from Programs in 14 states

2008

The Council of Chief State School Officers Washington, DC

Council of Chief State School Officers The Council of Chief State School Officers (CCSSO) is a nonpartisan, nationwide, nonprofit organization of public officials who head departments of elementary and secondary education in the states, the District of Columbia, the Department of Defense Education Activity, and five U.S. extra-state jurisdictions. CCSSO provides leadership, advocacy, and technical assistance on major educational issues. The Council seeks member consensus on major educational issues and expresses their views to civic and professional organizations, federal agencies, Congress, and the public. State Education Indicators The Council is a strong advocate for improving the quality and comparability of assessments and data systems to produce accurate indicators of the progress of our elementary and secondary schools. The CCSSO education indicators project is providing leadership in developing a system of state-by-state indicators of the condition of K–12 education. Indicators activities include collecting and reporting statistical indicators by state, tracking state policy changes, assisting with accountability systems, and conducting analyses of trends in education. CCSSO also works with states on studies and reports to analyze effects of state education policies and programs. This study is conducted under a grant to the Council of Chief State School Officers from National Science Foundation, Grant # REC 0438359. This CCSSO study is also possible because of the excellent cooperation and coordination by staff in each participating state department of education.

2008 Council of Chief State School Officers Rick Melmer (South Dakota), President T. Kenneth James (Arkansas), President-Elect Elizabeth Burmaster (Wisconsin) Past President Gene Wilhoit, Executive Director Rolf K. Blank, Director of Education Indicators Council of Chief State School Officers Attn: Publications One Massachusetts Ave., NW, Suite 700 Washington, DC 20001 202-336-7016 Fax: 202-408-8072 www.ccsso.org ISBN: 1-884037-25-9

Copyright © 2008 by the Council of Chief State School Officers. All rights reserved.

1

Does Teacher Professional Development Have Effects on Teaching and Learning? Analysis of Evaluation Findings from Programs for Mathematics and Science Teachers in 14 states

Rolf K. Blank Nina de las Alas Carlise Smith February 2008

Report prepared under a grant to the Council of Chief State School Officers from the National Science Foundation, Grant # REC 0438359: Project title: “Improving Evaluation of Professional Development in Mathematics and Science Education” Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

For further information, go to the Project Webpage http://www.ccsso.org/projects/improving_evaluation_of_professional_development

Council of Chief State School Officers, Washington, DC.

0

Table of Contents Page Summary of Findings.......................................................................................................... 1 Study Purpose ..................................................................................................................... 3 Phase I: Findings on Quality of Professional Development ............................................... 5 Phase II: Analysis of Evaluation Results across States ...................................................... 6 Student Outcomes ..................................................................................................12 Teacher Content Knowledge..................................................................................15 Instructional Practices............................................................................................19 Professional Development Designs with Measurable Effects ...............................21 Instruments Used in Evaluations ...........................................................................24 Conclusions and Recommendations ..................................................................................26

References and List of Evaluation Study Reports Reviewed ............................................29 Appendix: Contacts for Reports.........................................................................................34

2

1

Does Teacher Professional Development Have Effects on Teaching and Learning? Analysis of Evaluation Findings from Programs for Mathematics and Science Teachers in 14 states In 2005 the Council of Chief State School Officers (CCSSO) began a study of teacher professional development programs in mathematics and science through a grant from the National Science Foundation. States nominated professional development programs for the study, and to conduct the study, the CCSSO team has worked with state coordinators and local program directors and evaluators. The study is designed to assist education leaders in all states by providing a cross-state analysis of the quality of professional development programs and evaluations using a common rubric developed from recent research on program effectiveness. For further information on the study design and results, see http://www.ccsso.org/projects/improving_evaluation_of_professional_development

Summary of Findings CCSSO reviewed evaluation studies from 25 professional development programs for teachers of mathematics and science from programs nominated by 14 states. The evaluation study reports and papers served as the data sources for the present analysis and paper. The reports primarily address evaluation findings from professional development activities conducted during the period 2004 through 2007. Following are several key findings from the study: •









One-third of evaluation studies reported measurable effects of teacher professional development. Seven of the evaluation studies of teacher professional development reviewed by CCSSO reported measurable effects of the teacher development activities on subsequent student outcomes. A total of 10 of the studies reported measurable effects on increasing teacher content knowledge, and four studies reported measurable effects on instructional practices of teachers. Content focus plus sufficient time plus in-school component equals significant effects. The cross-program review of studies showed significant effects of professional development programs for teachers of math and science when the programs include focus on content knowledge in the math and science subject areas plus training and follow-up pedagogical content knowledge. The total time in professional development for the studies with significant effects was 50 hours or more. Purposeful evaluations yield measurable effects. The evidence from the CCSSO review of evaluation studies shows that one-third of the programs reviewed had well-developed evaluations that produced findings with measurable effects on student achievement or change in instructional practices. Our analysis of evaluation findings emphasized scientific study design, and these kinds of designs could have been implemented across more programs. Teacher vs. school-based professional development designs provide differing data on success. Many designs for professional development are based on selection through teacherbased, voluntary models. The use of teacher-based professional development makes important follow-up activities harder to schedule and implement, and alignment to school curriculum more difficult to accomplish. For evaluation, the use of student assessment scores and tracking change over time appear to be facilitated with use of the school-based model for professional development. Include outcome measures in allocation of evaluation resources. Smaller professional development projects typically cannot afford ambitious, multi-stage evaluations or research. Allocation of more funds to evaluation would mean fewer participating teachers or fewer resources for the program implementation. In the cross-state program review we observe that smaller programs typically had to choose a few measures and methods of evaluation.

1









Plan for use of data systems and experimental designs. Evaluations that will measure effects over time require access to data collection instruments or data systems, and advance planning with school officials. About one-fourth of program evaluations in the study did include comparison of a treatment group with a control group of teachers. State managers should consider evaluation designs that can be completed by linking data from state student assessments or local assessments with data on professional development for teachers. Link teacher knowledge gains to change in classroom practices. One type of evaluation finding identified in this review of studies showing promise for further use and expansion to other PD studies was measurement of change in teaching practices in the classroom. Four of the studies implemented well-tested instruments for comparing classroom practices across samples of teachers and classrooms. With advance planning, teachers and classrooms can be selected so that change in practices can be measured at the baseline point when teacher development begins and after implementation of activities and a period of implementation has been experienced. Use findings in program decisions. With the recent attention to scientific designs to provide measurement of impact of professional development on learning, we would like to see greater focus on how results from evaluations will be provided to decision-makers at specific points of time in the course of a project, and not long after the program activities have concluded. This is particularly important if a specific model is being considered for replication or expansion to other districts, schools, or additional teacher groups. Value partnerships for evaluation. Our analysis of evaluation findings across a number of programs and studies indicates that partnerships between higher education institutions and school districts have generally not added to the capacity for evaluation of professional development. For partnerships led by higher education institutions, the key partners with regard to data and measures for evaluation are local school district decision-makers and state education agency officials.

2

Study Purpose: Analyze current teacher professional development in relation to recent research In the present education policy environment, a high priority has been placed on improving teacher and teaching quality in U.S. schools. Standards-based educational improvement requires teachers to have deep knowledge of their subject and the pedagogy that is most effective for teaching the subject. States and school districts are charged with establishing and leading professional development programs, some with federal funding support, which will address major needs for improved preparation of teachers. Central to efforts to improve the quality of professional development is research-based evidence of effective programs and analysis of the characteristics of programs that make them effective. Current policies at national and state levels are attempting to address the need for improving the preparation of teachers through professional development programs that have been demonstrated through research to be effective. Research over the past decade has provided a base of knowledge about the characteristics of effective programs of teacher professional development in mathematics and science. Recent federal funding directed toward teacher professional development, such as No Child Left Behind (NCLB) Title II and the National Science Foundation (NSF) teacher enhancement programs has reflected findings of research in the 1990s indicating the characteristics of programs that are effective in improving teaching and learning. Effective math and science professional development has been found to have several common characteristics, including focus on content knowledge and skills of teachers, coherence with state and district standards for learning, sustained over time, and active methods of teacher learning of practices (see Banilower, Boyd, Pasley, & Weiss, 2006; Birman & Porter, 2002; Corcoran & Foley, 2003; Cohen & Ball, 1999; Cohen & Hill, 2001; Frechtling, 2001; Garet, Porter, Desimone, Birman, Herman & Yoon, 1999; Hiebert, 1999; Loucks-Horsley, Hewson, Love & Stiles, 1998; Kennedy, 1999; Weiss, Banilower, McMahon, & Smith, 2001). However, while a decade of research studies were finding strong evidence of what works, the data from large-scale national studies showed that most professional development provided to teachers did not meet these quality characteristics (Garet, Porter, Desimone, Birman & Yoon 2001; Desimone, Porter, Garet, Yoon & Birman, 2002; Corcoran & Foley, 2003). Thus in large part educators may know what kinds of programs should be developed and implemented based on research findings, but there are a variety of organizational, policy, and structural factors in education that have inhibited major change in practice. Another emphasis of federal programs support since 2001 has been an emphasis on scientifically-based research and evaluation. State and local program grantees are asked to base their program designs for professional development on research evidence. Additionally, program designs are strongly encouraged to implement evaluation methods for professional development activities that use experimental designs, particularly randomized comparison trials that measure the outcomes of professional development using treatment and comparison groups. In many states and districts, program designers have worked to partner with researchers in universities, educational research organizations, or consultants to develop evaluations that are more robust, include validated measures and instruments, and track the effects of professional development efforts with teachers over time. The CCSSO study on “Improving Evaluation of Professional Development in Mathematics and Science Education,” supported by a grant from National Science Foundation, was designed to analyze the quality of a voluntary sample of professional development programs, as compared to the research, and to analyze the outcomes and evidence from the evaluation studies. Our overall goal is provide feedback to state education leaders about the degree to which this set of nominated professional development programs from a range of states meet criteria of high quality professional development established from research. We also wanted to identify common findings across programs on the effects of professional development on teachers and on their students.

3

State Leader Needs. State Education Agencies (SEAs) and state leaders have several roles in providing leadership with professional development for teachers. Each of these roles can benefit from research and evaluation findings concerning the program characteristics that have positive impact on improving teacher knowledge and skills and improving the quality of teaching and learning in classrooms: • •





States write and set policies requiring teacher professional development, and in most states that have professional development requirements for re-certification the policies have a major effect on the types of professional development that are offered SEAs manage federal and state funds that are designated for teacher professional development and staff members often have responsibility for determining the size, scope, focus, and types of professional development teachers will receive, and many programs have specific evaluation requirements. Under the NCLB Title IIB funding for math and science states are responsible for awarding Math Science Partnership (MSP) local grants and each grantee must provide evaluations and reports on the effects of the MSP funding. As this program has grown since 2002, many states are taking a larger role in providing a statewide program evaluation design for grantees State education specialists who manage professional development or a curriculum subject area often provide recommendations to local districts and others concerning their professional development programs and strategies—including programs during scheduled in-service days, evening/weekend or summer programs, and teacher course credit options SEAs manage other areas that have significant and important implications for teacher professional development, including: development and implementation of state content standards for student learning, review and recommend curriculum materials and textbooks, and standards for teacher licensure and certification, and school improvement strategies and designs for lowperforming schools and districts

Use of the Study Results. As a result of the important roles of state leaders in shaping and leading professional development in math and science education, CCSSO has undertaken the current cross-state study of the characteristics of high-quality, effective professional development. Two key expected uses of the study results for state leaders are: •



Guidance on how to use findings from research on program quality—in designing, selecting, or leading professional development programs supported by state agencies or in advising local agencies. State leaders will receive specific examples of the characteristics of professional development that will match what we know from research, and we can use evidence and examples from the current programs that are reviewed; Assistance with evaluation designs, tools, and use of evaluations—to improve the role of states in establishing designs and principles for evaluation and expectations for how the evaluation results will be used. All of the programs nominated by states for the CCSSO study have methods of evaluating effects of programs and activities, but the quality and appropriateness of methods used vary widely. The study provides a review of current methods and recommendations on how states can strengthen their evaluations and improve the usefulness of the evidence.

4

Phase I: Findings on Quality of Professional Development In the first phase of the study, CCSSO conducted a systematic review of 25 teacher professional development programs submitted by 14 participating states. The review was based on the program documents from proposals, designs, and initial reports. The review process was conducted through a PD Program Review Rubric and Guide, developed through the study, and teams of experts rated each of the 25 programs. The analysis addressed two questions: • •

What is the quality of professional development across the nominated sample of programs, and what is the extent of variation in quality? What are the main program characteristics contributing to high ratings for quality that can be identified and replicated in future program design and development?

As a group, the programs could be considered representative of the current leading efforts to improve the teaching of math and science in public schools. The findings from the CCSSO study provide a way to analyze the status and prospects for math and science teacher professional development, and particularly initiatives supported through federal and state funding (see Blank, de las Alas, & Smith, 2007; http://www.ccsso.org/projects/improving_evaluation_of_professional_development). The findings from first Phase program review can be summarized as follows: Content Focus: The CCSSO analysis (completed at the end of 2006) found that current leading professional development is providing content knowledge development for teachers in math and science, especially for elementary and middle grades teachers. In 22 of 25 programs reviewed, the activities were rated as significantly focused on content knowledge in math or science. Additionally, a majority of programs reviewed were rated positively for providing important pedagogical content knowledge in math or science for teachers. Active Learning: The professional development activities use active methods of learning for teachers in a large majority of programs. In comparison to the findings from research on professional development in the mid-1990s, the sample of programs in this study were surprising in the prevalence of active roles by teachers, including developing and presenting sample lessons, use of coaching and mentoring, developing new lessons or assessments, and interaction among teachers about ways to improve their practice. Collective Participation: While most of the 25 reviewed programs did organize teachers and activities by common subject area and grade level, only a minority of the sample programs focused on delivery of professional development to teachers through a school-based strategy where teachers were learning with their school colleagues. The predominant organizing pattern was to plan teacher development for a treatment group drawn from a large number of schools and districts, with only a small number of teachers from each school. Coherence: In almost all programs examined,, the reviewers found a description of how the program was designed to be aligned to state content standards. Additionally, a majority of program materials described how the development was consistent with local curriculum or with curriculum materials teachers were intended to use. Sufficient Time: The average time for professional development activities including follow-up work in schools was found to be significantly greater than the typical math and science professional development documented and described in the mid-1990s. Evaluation: The reviewed programs included evaluation designs with a number of evaluation objectives and tools. The CCSSO review covered four specific evaluation objectives, and a majority of programs included at least one measure in each objective: (a) quality of implementation of development activities; (b) gains in teacher knowledge; (c) change in classroom practices, and (d) increase in student achievement. Thus, in general, the programs were very ambitious in emphasizing methods of evaluation.

5

Phase II: Analysis of Evaluation Results across States The second phase of the study, the subject of the present report, is an analysis of outcomes and findings from the program evaluation reports. CCSSO has analyzed data and findings from evaluations conducted for the 25 programs in our study. This second study report addresses the following four questions about the results of our analysis of evaluations of professional development: • What evaluations were completed and what findings were reported? • What were the types of major findings from the evaluations? How were they measured? What

measures of outcomes were used in the PD evaluations?

• What conclusions can be drawn about the adequacy and usefulness of the evaluations and

reports? What are the cross-report recommendations that are useful to state leaders and evaluators?

CCSSO has compiled and analyzed the evaluation studies and reports completed and provided us as of September 2007. Thus the period of evaluation findings that were included in our cross-state analysis were prepared and reported from Spring 2005 through Spring 2007. Study Evaluations Completed and Reported Initial Evaluation Designs. In the first phase of the CCSSO study, we analyzed the designs for evaluation of professional development and we categorized the types of evaluations intended to be used. The 25 programs in our study were identified in Spring 2005, and the descriptions of intended professional development activities and evaluation designs were reviewed by CCSSO in 2005 and 2006. The analysis of evaluation designs of the teacher professional development programs was conducted using four categories of outcomes. These categories are consistent with the logic model for the CCSSO study design (CCSSO, 2005) based on our review of the research: (a) (b) (c) (d)

quality of implementation of professional development activities gain in content knowledge and pedagogy skills of teachers change in instructional practices or curriculum that is taught improvement in student achievement

As shown in Figure 1, almost all of the programs had planned to carry out evaluations intended to assess professional development across at least three of the four categories of outcomes. Seven had evaluation designs that cover all four categories and other outcomes. These programs utilize multiple measures to report to their stakeholders on how the program is being implemented and the degree to which the program activities are having effects on teaching and learning. In several cases, the programs also track other outcomes specific to the program.

6

Figure 1 - Summary of Intended Program Evaluation Design 25

# of Programs

20

15

10

5

0

Quality of PD Activities

Teacher Content Knowledge

Instructional Practices or Curriculum

Student Achievement

Other Outcomes

Evaluation Reports and Studies. In Fall 2006, CCSSO requested evaluation reports from all of the sample programs that included specific evaluation designs for professional development, and the information was summarized above. In Spring 2007, CCSSO communicated a second request for evaluation reports and studies to all of the participating program directors and evaluators. As of September 2007, CCSSO received a total 41 evaluation reports or studies, and thus a majority of the programs provided multiple reports. These documents served as the data sources for the present analysis and paper. The reports address evaluation findings from professional development activities conducted during the period 2004 through 2007. Reports vary in analyzing data for one, two, or three years of activities. Table 1 lists the evaluation reports received across the programs in our study. The columns of the table indicate the type of data reported by evaluation category (e.g., PD activities— measured by teacher survey). The column on the far right indicates the number of teachers that involved in the professional development and their subject and grade level. The 41 reports include a range of outcomes data and descriptive analyses, and some of the evaluation evidence pertains to program outcomes other than for professional development, including pre-service, curriculum materials, and partnership development. Many of the evaluation reports included formative, descriptive data which were intended as internal evaluation information primarily used by the study team and institutional administrators. In the table, the CCSSO study team identified the data and findings from four types of evaluation outcomes from professional development. These data and findings were the focus of our cross-state and cross-study analysis.

7

Making Math More Meaningful – M4 (IN)

iCATS: Beyond the Textbook (IN)

Developing Mathematical Thinking - DMT (ID)

Launch II (FL)

Science Lead Teacher (DE) Science Coalition Courses (DE)

Teacher Quality Enhancement/Secondary Teacher Enhancement Project - TQE/STEP CO)

Rocky Mountain Middle School (CO)

Northeast Front Range (CO)

Program

• Students with advanced courses by district • Student achievement by year for student of new Gr. 610 teachers • Reported on Baseline Year 2005

• May 2007 - Yr 3 Report

• 2005-06 report

• July 2006 - Year 3 Report

• 2005 - Year 2 Report

• May 2004 - Year 1 Report

• 2006

• 2005

• September 2006 (External Evaluator)

• Reported change in ISTEP student achievement (Grades 3-8) from 2003-04 to 2004-05 of students from treatment math teachers, no comparison group

• Direct Math Assessment (K2) • Achievement gains of students of treatment teachers vs. gains of students of control teachers • Reported pre-post tests for Gr. 4 & 5, Year 1 & Year 2, Gr. 3, 4, & 5 in Year 3

• ISAT planned

• Reported on 2005 to 2006 school mean in math and in science vs. district mean

- None -

• October 2000

• 2006

- None -

• Report, no date

• October 2006 - Year 2 Report

• December 2005 - Year 1 Report

• May 2006 - Yr 2 Report

• Change from 2004 to 2005, 2005 to 2006 • Treatment vs. Control Groups • No student achievement scores reported

• Student achievement linked to treatment teachers

• November 2004 - Yr 1 Report

• March 2006 - Yr 2 Report (Jan.–Dec. 2005) • March 2007 - Yr 3 Report (Jan.–Dec. 2006)

Student Achievement

Report Date

Table 1: Program Evaluation Reports by Objective

8

• PD is associated with the implementation of Everyday Math curriculum

• Based on interviews w/ teachers

- None -

- None -

• Measured through survey of teachers on PD activities

- None -

- None -

• Retention rate of new teachers/yr.

- None -

- None -

• Measured through teacher math knowledge inventory • Analyzed pre-post gains

• Measured through prepost teacher knowledge test

- None -

- None -

• Measured through exams of new teachers

• Analyzed pre-post gains per course reported • Measured through PRAXIS, PLACE

• Measured through teacher knowledge test

• Measured through teacher surveys • Measured through survey of IHE staff • Measured through survey of new teachers

• Measured through prepost test by course

Teacher Knowledge

• Measured through teacher survey by course

PD Activities

• Based on observations of 26 teachers

- None -

• Measured through SEC surveys on instructional practices

• Measured through survey of instructional practices

- None -

- None -

- None -

• Measured through teacher survey on instructional practices

• Examined impact through focus group

• Measured through teach survey on student

Instructional Practices

• Elementary teachers: approx. 36 per summer institute, 57 per inschool sessions

• Approx. 57 teachers/yr.

• 60 elementary teachers

• 161 elementary & middle teachers

- None -

- None -

• 364 teacher leaders

• 175 math & science teachers

• 136 treatment teachers/yr.

# of Teachers

Pursuing Excellence in Middle School Math & Science – Nash-Rocky Mount (NC)

Consortium for New Explorations in Coherent Teacher Education CONNECT-ED (NJ)

Teachers as Leaders and Learners - TaLL (NJ)

Coalition for Higher Standards Math Partnership Program Lesley/MassInsight (MA)

EduTron (MA)

Mathematics Access and Teaching in High Schools - MATHS (ME)

High School Mathematics Alliance Grant (KY) Math/Science Teachers Engaged as Mentors MS*TEAMS (KY)

Program

Table 1 – continued

• October 2006 [Data table] • October 2006 (2004-05) (External Evaluator) • October 2006 final report (2004-05) • January 2007 report

• July 2005 Report

• October 2004, pilot project

• October 2006

• October 2005

• January 2006 (External Evaluator)

• 2006

• Fall 2006 Addendum

• 2006

• August 2006 (2004-06 implementation period)

• September 2005 - Year 1 Report

• Change in % of students at proficiency or beyond by grade level (grades 6-8) from 2003-04 to 2004-05 in end-ofgrade math assessment • % of students at proficiency in 2005-06 (preliminary data) for Gr. 6 in end-of-grade and alternative math assessments

- None -

• Math teachers vs. control teachers, by school • Student achievement linked to teachers • Analyzing change from 2004 to 2005, and 2005 to 2006 in Terra Nova (dist test) • Treatment vs. control groups

• Reported on student achievement for 2004, baseline year

• Student achievement district level change from 2003 to 2006

9

• Based on teacher focus group interviews

• Based on interviews, observations

• Measured through teacher survey on practices and attitudes

- None -

• Pre-treatment assessments of participant teachers determined type of course offered

- None -

• Measured through Gr. 9 Algebra Concepts Test • Student interviews & focus groups, classroom observation • Linked to treatment teachers • Analyzing student achievement change from 2005 to 2007

• Analysis of teacher assignments for increased math understanding using 5 levels • PRAXIS

- None -

• Based on grades in math courses

• Treatment teachers vs. control teachers

• Measured through teacher knowledge test

• Pre-post teacher knowledge test

• Measured through MA teacher licensing test (MTEL)

- None -

- None -

• Descriptive report of evaluator • Interviews with teachers

- None -

• 2006-Year 2 Report

Teacher Knowledge - None -

PD Activities - None -

- None -

Student Achievement

• No report

Report date

- None -

- None -

• Grades

• Measures on teacher survey

• Based on classroom observations

• Based on interviews of coaches, teachers

- None -

• Reported through student focus groups

• Level of cognitive activity

• Based on classroom observation, analyzing types of instruction

- None -

- None -

Instructional Practices

• 35 teachers and coaches all levels

• 40 treatment teachers in 10 design teams

• 20 middle school math teachers treatment group

• 100 elementary & middle school math teachers as coaches in 4 school districts

• 127 elem/mid school math teachers in 3 districts • 40 elementary & middle school math teachers in last course cycle (Dec.-Apr.)

• 100 middle school & high school math teachers

• Learning teams

• 21 teachers

- None -

# of Teachers

Number of Evaluation Reports

Mathematics and Science Coaching Initiative – South Carolina Coaching Initiative (SC) Northwest Wisconsin Partnership for Mathematics and Science Education – NW Wisconsin (WI) Wisconsin Academy Staff Development Initiative Retention and Renewal – R & R (WI)

Oregon Mathematics Leadership Institute OMLI (OR)

Willamette Valley Watershed Project (OR)

41 total

• June 2005, 2004-05-Year 2 Exec. Summary

• n.d.

• December 2005

• 2007 report on Fall 2003Spring 2006

• April 2007

• May 2006

• October 2006 [Part 2: Narrative Responses]

• October 2006

• September 2005

• 2005

• May 2006

Comprehensive, DataBased Professional Development for Middle School Mathematics Teachers (OH)

High Desert (OR)

• June 2006

Report date

BreakThrough Mathematics (OH)

Program

Table 1 – continued

20 reports with student achievement data 11 reports with treatment vs. control comparison

• No student achievement data reported

• No student achievement data reported

• No student achievement data reported

• Reported on student achievement in science based on test specific to the project. • Compared treatment & control group, using pre-post testing • Reported on % of students who met or exceeded the math standards on state assessments between 200405 & 2005-06 • Reported on results from student discourse observation

10

14 on PD activities

- None -

• Measured through interviews

• Measured through teacher survey

- None -

- None -

- None -

• Teacher participation rate

• Reported on student achievement change and comparing treatment vs. control groups, based on results in 2004 to 2005

24 on teacher knowledge

- None -

• Measured through teacher self-rating of knowledge gains

- None -

• Measured through teacher content knowledge test, reported pre-post results

• Measured through teacher survey

• Measured through teacher knowledge test linked to student achievement test

• Measured treatment teacher knowledge through DTAMS • Compared PRAXIS results between treatment vs. control teachers

- None -

- None -

• Measured through teacher follow-up survey on perceptions, satisfaction on program • Based on evaluators/program implementers' observations (no protocol)

Teacher Knowledge

PD Activities

• Percent proficient by school in math and/or science, baseline from 2004-05 results

- None -

Student Achievement

13 on instructional practices

- None -

- None -

• Measured through teacher survey to track instructional change

• Based on classroom observation

• Measured through instructional practice survey on sample of teachers

• Analyzed lesson plans

• Needs assessment

• Measured through SEC, RTOP

• Teacher mentors

• 40 teachers/year

• 30 middle school math and science teachers

• 600+ teachers

• 33 science and math teacher coaches

• 280 School Leadership Team members (2 teachers & 1 administrator per team)

• 15 treatment teachers, 15 control teachers

• 24 elementary math teachers

- None -

• 194 teachers taking online math course

• Measured through survey on applying newly acquired knowledge/skills to practices • Classroom observations and interviews by coaches

# of Teachers

Instructional Practices

Implementation of PD. A total of 14 reports included evaluation data on the quality of how professional development activities were implemented (i.e., extent of teacher participation, fidelity of implementation of PD design, and response of teachers to the professional development). All of the evaluation reports include reports on how the activities were implemented and descriptions of the activities. Some of the evaluation studies heavily focused on the use of information as formative evaluations to assist decisionmakers and leaders to assess how the program is operating and how improvements can be made. However, some of the data provide outcome findings on the quality of implementation of the professional development. Gains in teacher knowledge. Twenty-four of the 41 evaluation reports included data and findings regarding gains in teacher knowledge related to the professional development activities being evaluated. Further analysis of these data across sites will allow us to determine whether there are common factors in producing gains in knowledge and how the several knowledge tests worked to measure gains. Change in classroom instructional practices related to the professional development. Thirteen reports included data on instructional practices of the teachers participating in professional development, allowing program evaluators to determine if there was change in instructional practices of participating teachers. A variety of measures and tools were found to be in use. Improvement in student achievement related to teacher professional development. Twenty of the evaluation reports included data on student achievement for teachers, schools, and districts in which teachers were involved with the professional development activities. A major question for our CCSSO study was the degree to which student achievement gains could be attributed to the treatment of professional development. Sixteen evaluation reports included student achievement trends for at least two years. A small number tracked achievement gains for more than two years. In sum, the evaluation reports do not cover as many of the design objectives as were outlined and intended in the program proposals and initial descriptions. The reports received include findings from slightly more than half of the programs in any one category, or significantly fewer results than originally planned. It is possible that additional evaluation reports and studies will be released by programs.

Analysis of Major Findings Identified in Evaluation Reports CCSSO staff reviewed the findings from the evaluation reports submitted from the programs in the study. The goal of CCSSO’s analysis was to identify findings from the reports that are based on measurable effects of teacher professional development. A set of criteria was established for determining measurable effects under each of the evaluation categories. This approach to identifying cross-state findings allows CCSSO to highlight evaluation studies that can be held as examples of professional development that provide clear, scientific evidence of impact on teaching and learning. The studies for which we found measurable effects of professional development are grouped by type of outcome. We provide the core evidence in a table that allows the reader to review findings across studies and to understand the basis for selection of measurable effects.

11

Student Outcomes We display in Table 2 the findings of effects of teacher professional development on student outcomes. Three primary criteria were used to determine findings that can be classified as demonstrating effects: a) Finding of effect on student outcome is supported by statistical significance of change linked to the treatment teachers b) Finding is substantively important, i.e., an educationally significant change c) Measure of student outcomes is reliable and valid for the evaluation purpose Table 2: Evaluations Reporting Measurable Effects of Teacher Professional Development on Student Outcomes Evaluation Report, PD META Associates. (2007), NE Front Range-CO Schmidt, D. L. (2006), Launch II-FL EvansvilleVanderburgh Schools. (2006), iCATS-IN Grip, R. S. (2006), TaLL-NJ Hansen, J. B. (2006), Willamette Valley-OR Weaver, D. (2007), OMLI-OR

Niess, M. L. (2005), High Desert-OR

Outcome Measure

Study Design

Student gains on CSAP Math for two years

Quasiexperimental Pre-/posttests

Student gains on FCAT elementary math, science assessment ICAT math gains for Grades 3, 4, and 5 for 2 years

QuasiExperimental Pre-/posttests

Gains on districtwide Terra Nova test in math & science Gains on Student Science Achievement Level Test (developed with Northwest Evaluation Assoc.) Change in student discourse as observed thru teacher surveys, Student Discourse Protocol Student gains on state math assessment for elementary & middle grades

Quasiexperimental Pre-/posttests

Pre-/posttests School-byschool analysis Quasiexperimental Pre-/posttests

Treatment/ Comparison Matched comparison group of teachers

Teacher PD whole school vs. weighted district level performance Treatment students vs. students participated in direct teaching environment Treatment vs. non-treatment teachers in same school Matched comparison group of teachers

Finding CSAP state assessment results show statistical significant mean gain of 13.5 points for students of participant teachers, with 70% improved in scale score, and higher gains than students of comparison teachers (higher initial scores). FCAT results show statistically significant gains on FCAT math in Grades 4-5 for treatment schools (+2.1%) vs. comparison (+.8%). Statistically significant gains on ICAT Math for Grades 3 – 5 students of teachers receiving PD with Educational Technology (gain scores 4 points higher than comparison students). Over two years, significant gains on Terra Nova test for students of teachers in PD (in math 7-pt. higher gain; in science 2-pt. higher gain). Treatment group outperformed the comparison group at grades 8, 9, but not grades 6, 7. Effect size moderate (.106) across grades.

Quasiexperimental Pre-/posttests

Teachers in treatment matched to nonparticipant comparisons

Student discourse in classes of teachers in treatment group significantly greater than comparison teacher classes.

Quasiexperimental Pre-/posttests

Matched comparison group of teachers

From 2004 to 2005 elementary treatment group showed statistical gains in math achievement (214 to 220, p

Suggest Documents