DRAFT ‐‐ This document was created for discussion purposes to facilitate gathering input to inform the creation of evaluation rubrics per the requirements of EC 52064.5. This is intended to serve as a conceptual example to inform the creation of a draft for future review and discussion.
Evaluation Rubrics User Acceptance Testing Reviewer Guidance December 2015 Thank you for your participation in the State Board of Education and WestEd’s user acceptance testing (UAT) of the Local Control Funding Formula (LCFF) evaluation rubrics. The feedback shared as part of the initial phase has helped inform improvements to the content and design of the evaluation rubrics. Last week’s passage of the Every Student Succeeds Act (ESSA) presents an unexpected opportunity to support alignment between local, state, and federal accountability and improvement. Under ESSA each state designs an accountability system. The broadening of indicators in ESSA used for accountability is consistent with the concept and approach under development for the evaluation rubrics. The evaluation rubrics process continues and will be coordinated with the development of new approaches to local, state, and federal accountability. Following is an excerpt from Education Week1 that explains the ESSA accountability requirements. States would still have to submit accountability plans to the Education Department. These new ESSA plans would start in the 2017-18 school year. Goals:
No more expectation that states get all students to proficiency by the 2013-14 school year, as under NCLB Classic. (That ship has sailed, anyway.)
Instead, states can pick their own goals, both a big long-term goal, and smaller, interim goals. These goals must address: proficiency on tests, English-language proficiency, and graduation rates.
Goals have to set an expectation that all groups that are furthest behind close gaps in achievement and graduation rates.
What kinds of schools will states have to focus on?
States have to identify and intervene in the bottom 5 percent of performers, an idea borrowed from waivers. These schools have to be identified at least once every three years. (That's something many states already do under waivers. And some, like Massachusetts, do it every single year.)
States have to identify and intervene in high schools where the graduation rate is 67 percent or less.
States, with districts, have to identify schools where subgroup students are struggling.
What do these accountability systems have to consider? The list of "indicators" is a little different for elementary and middle schools vs. high schools. Systems for Elementary and Middle Schools:
States need to incorporate a jumble of at least four indicators into their accountability systems.
That includes three academic indicators: proficiency on state tests, English-language proficiency, plus some other academic factor that can be broken out by subgroup. (That could be growth on state tests,
1
Retrieved from: http://blogs.edweek.org/edweek/campaign-k12/2015/11/esea_reauthorization_the_every.html?r=1749565660&preview=1 (December 13, 2015)
1
DRAFT ‐‐ This document was created for discussion purposes to facilitate gathering input to inform the creation of evaluation rubrics per the requirements of EC 52064.5. This is intended to serve as a conceptual example to inform the creation of a draft for future review and discussion.
so that states would have a mix of both growth and achievement in their systems, as many already do under waivers.)
And, in a big new twist, states must add at least one, additional indicator of a very different kind into the mix. Possibilities include: student engagement, educator engagement, access to and completion of advanced coursework, post-secondary readiness, school climate/safety, or whatever else the state thinks makes sense. Importantly, though, this indicator has to be disaggregated by subgroup.
States also have to somehow figure in participation rates on state tests. (Schools with less than 95 percent participation are supposed to have that included, somehow.) But participation rate is a standalone factor, not a separate indicator on its own.
Systems for high schools:
Basically the same set of indicators, except that graduation rates have to be part of the mix. They take the place of a second academic indicator.
So to recap, that means for high schools: proficiency on tests, English-language proficiency, graduation rates, plus at least one other indicator that focuses a little more on whether students have the opportunity to learn, or are ready for post-secondary work.
And also, test participation has to be incorporated in some way. (But it's a standalone factor, not a separate indicator like test, grad rates, or those non-academic factors).
How much do each of these indicators have to count? That will be largely up to states, but the academic factors (tests, graduation rates, etc.) have to count "much" more as a group than the indicators that get at students' opportunity to learn and post-secondary readiness.
Given the references to graduation rate in both the LCFF state priorities and ESSA, the Phase II user acceptance testing builds on the graduation rate example provided in Phase I. A number of questions were asked regarding the method for assigning the color-coded classification for improvement and outcome shared in Phase I. Improvement
Outcome Very High
High
Intermediate
Low
Very Low
Improved Significantly
Excellent
Good
Good
Good
Emerging
Improved
Excellent
Good
Good
Emerging
Issue
Maintained
Excellent
Good
Emerging
Issue
Concern
Good
Emerging
Issue
Issue
Concern
Emerging
Issue
Issue
Concern
Concern
Declined Declined Significantly
The following attachment provides an analysis used to develop a potential recommendation for a graduation rate quality standard. As you review the example, please consider the following questions: Does the color-based classification approach support local accountability? Continuous improvement? Please explain. 2
DRAFT ‐‐ This document was created for discussion purposes to facilitate gathering input to inform the creation of evaluation rubrics per the requirements of EC 52064.5. This is intended to serve as a conceptual example to inform the creation of a draft for future review and discussion.
What aspect(s) of the rationale/analysis used to determine the quality standard is most compelling? Based on what you know about ESSA, should local, state, and federal standards used for accountability be the same or different? In other words, should the quality standards for the evaluation rubrics be the same as federal accountability standard? Please explain. What additional advice or questions do you have at this time?
Reviewer Instructions There are two steps to the review process: 1. Please review the UAT Phase II packet and share the information in the packet with staff that are involved with planning, implementation, and monitoring related to the state priorities. 2. During the week of December 14-18, State Board of Education and WestEd staff will conduct follow-up calls to review feedback from your review of the content examples. Please feel free to invite those that were involved in the review process from your organization to participate. The calls are expected to take 35-45 minutes.
3
DRAFT ‐‐ This document was created for discussion purposes to facilitate gathering input to inform the creation of evaluation rubrics per the requirements of EC 52064.5. This is intended to serve as a conceptual example to inform the creation of a draft for future review and discussion.
Attachment: Analysis of Graduation Rate to Inform Quality Standard The following is provided as an example of analysis developed to create a recommendation for a graduation rate quality standard that could be referenced in the evaluation rubrics. The information that precedes the recommendation at the end of the attachment is background and analysis and is not expected to serve as content for the evaluation rubrics. It does however, explain a possible approach to deriving the quality standard.
Metric: Graduation Rate Background: In order to graduate from California public high schools, students must complete specified state and local graduation requirements. Local school districts have the authority and responsibility for establishing high school graduation requirements. These requirements vary among school districts. However, California Education Code Section 51225.3 specifies that students must pass a minimum set of required courses and an exit examination. These requirements should be viewed as minimums and support for the regulations specified by the local school boards.2 Since 2009-10, the CDE has reported 4-year cohort graduate rates, which identify a "cohort" or group of students that could potentially graduate during a 4-year time period (grade 9 through grade 12). This cohort is then "adjusted" by adding students who transfer in to the cohort and subtracting the students who transferred to another school that offers a high school diploma, emigrated to another county, or died during the years covered by the cohort rate. Students who drop out during the four year period remain in the adjusted cohort, as well as students who complete 12th grade and exit the educational system without graduating. Students who take longer than four years to graduate or remain enrolled after four years are also included as part of the cohort. However students from the cohort who (1) pass the General Education Development (GED) test, (2) complete requirements necessary to obtain a special education certificate of completion, or (3) remain enrolled in the 9-12 instructional system without a high school diploma are not included in calculations for either the cohort graduation or cohort dropout rates. These groups of students receive separate completer rates (GED Completer Rate, Special Education Completer Rate, and Still Enrolled Completer Rate). Thus, the cohort graduation rate and the cohort dropout rate will not sum to 100% when one or more of these other completer rates exist within the cohort.3 California’s current federal accountability standard/target for graduation rate is 90%. X Local Education Agency Applies to: K-8 Grade Levels
X X
School 9-12
X Subgroup X K-12
Data Management: Data reported to and managed by CDE 2
Retrieved from: http://www.cde.ca.gov/ci/gs/hs/hsgrgen.asp (December 6, 2015) Cohort Graduation explanation adapted from Ed‐Data: http://www.ed‐data.org/article/Student‐Level‐Data‐and‐ Dropout_Graduation‐Rates (retrieved December 6, 2015) 3
4
DRAFT ‐‐ This document was created for discussion purposes to facilitate gathering input to inform the creation of evaluation rubrics per the requirements of EC 52064.5. This is intended to serve as a conceptual example to inform the creation of a draft for future review and discussion.
Analysis: Graduation rates are a commonly collected metric with most states using comparable definitions. California has shown a steady increase in graduation rates over time, yet gaps between subgroups persist. The table below shows California’s 4-year cohort graduation rate from 2009-10 through 2013-14. 2009-10 2010-11 2011-12 2012-13 2013-14 74.7% 77.1% 78.9% 80.4% 81.0% All Students 71.4% 73.7% 75.7% 76.6% Hispanic 68.1% 68.5% 72.4% 72.8% 70.6% American Indian 67.3% 90.3% 91.1% 91.6% 92.4% Asian 89.0% 74.9% 77.0% 78.4% 80.4% Pacific Islander 72.3% 89.9% 90.8% 91.6% 92.2% Filipino 87.4% 62.8% 66.0% 68.1% 68.2% African American 60.5% 85.7% 86.6% 87.7% 87.6% White 83.5% 71.1% 73.0% 74.8% 75.6% Low Income 68.0% 61.5% 62.0% 63.1% 65.4% English Learner 56.4% N/A N/A N/A N/A N/A Foster Youth 59.5% 61.1% 61.9% 62.3% Student with Disability 56.7% The following charts show how district-level graduation results vary across the state. The figure below shows three year average graduation rate (2011-12 to 2013-14) and the change in percentile points during this time for all districts reporting a graduation rate during this period of time. Graduation Rate Outcomes and Improvements, All Students (2011-12 to 2013-14)
As shown, most districts overall graduation rate held steady or improved slightly. As one would expect, greater amounts of improvement were observed by those with lower average graduation rates. 5
DRAFT ‐‐ This document was created for discussion purposes to facilitate gathering input to inform the creation of evaluation rubrics per the requirements of EC 52064.5. This is intended to serve as a conceptual example to inform the creation of a draft for future review and discussion.
Similar trends were observed for subgroups. Below is a figure showing the data for the African-American (red, triangle data points) and English Learners (green, diamond data points). Graduation Rate Outcomes and Improvements, African-American Students (2011-12 to 2013-14)
Graduation Rate Outcomes and Improvements, English Learner Students (2011-12 to 2013-14)
6
DRAFT ‐‐ This document was created for discussion purposes to facilitate gathering input to inform the creation of evaluation rubrics per the requirements of EC 52064.5. This is intended to serve as a conceptual example to inform the creation of a draft for future review and discussion.
Based on the above three-year averages and three-year improvement calculations, the table below shows the distribution of results by decile.4 Percentile Outcome – 3Growth – 3 Year year Average Percentile Change 90th 96.1% 6.5% 80th 94.3% 3.9% th 70 92.3% 2.3% th 60 90.7% 1.4% 50th 88.8% 0.7% th 40 86.4% -0.2% 30th 83.3% -1.2% th 20 78.7% -2.8% 10th 44.8% and below -6.2% California’s cohort graduation rates are close to the national average with progress made in recent years to narrow the gap. Below are data from the National Center on Education Statistics.5 2010-11 2011-12 2012-13 77.1% 78.9% 80.4% California, All Students 79.0% 80.0% 81.4% National Average 87.0% 87.4% 88.2% National, Average of Top 10 States Approximately half of California’s LEAs are graduating students at a rate at or above the current national average for the 10 highest performance states. There are also examples from other states and large education systems of standards for cohort graduation rates. For example, Ohio’s accountability report card issues letter grades based on performance levels – greater than 89% = A/B, below 79% = F. The Ohio letter grades would correspond to the deciles shown for California as follows Ohio A/B level = California’s 50th percentile decile and above; F level would be the bottom 20th percentile and below. Chicago Public Schools’ (CPS) system assigns points using a five-point scale with the following point assignments: CPS Graduation Scale and Points 85% or above = 5 points 75-84.9% = 4 points 65-74.9% = 3 points 55-64.9% = 2 points under 55% = 1 point
California Graduation Comparison 40th percentile and above 20th – 40th percentile 10th -20th percentile 10th percentile and below
4
To calculate the deciles, the three‐year graduation rate averages and percentile change were rank ordered from lowest to highest. The results were then divided into 10 equally sized groups or bands. Note that the deciles reflect the distribution of the results. Because graduation rates are clustered or skewed, the 50th percentile point falls above the average. 5 NCES.
7
DRAFT ‐‐ This document was created for discussion purposes to facilitate gathering input to inform the creation of evaluation rubrics per the requirements of EC 52064.5. This is intended to serve as a conceptual example to inform the creation of a draft for future review and discussion.
Recommended Definition for the Evaluation Rubrics: As specified in the LCAP (CCR, Title 5 XXXX) high school graduation rates shall be calculated as follows: (1) The number of cohort members who earned a regular high school diploma [or earned an adult education high school diploma or passed the California High School Proficiency Exam] by the end of year 4 in the cohort where “cohort” is defined as the number of first-time grade 9 pupils in year 1 (starting cohort) plus pupils who transfer in, minus pupils who transfer out, emigrate, or die during school years 1, 2, 3, and 4. (2) The total number of cohort members. (3) Divide (1) by (2). Recommended Evaluation Rubrics Standard: California compares well against other states and national averages. Adjust the current policy of 90%, to correspond to the 60th percentile as the standard for graduation rate, classified as “Very High and High.” Based on current data, approximately 40% of LEAs would meet or exceed this standard. Very High High Intermediate Low Very Low 96.1% or 90.7 to 83.3 to 83.2 to 78.6% or above 96.0% 90.6% 78.7% below Outcome 6.5% or 1.4% to -1.2% to -1.3 to -2.9% or Improvement above 6.4% 1.3% -2.8% below Evaluation Excellent Good Emerging Issue Concern Rubrics Designation
8
