Improving the Targeting of Treatment: Emerging Research on Postsecondary Math Placement Policies Quantitative Leap! Webinar Series: Webinar 2
June 8, 2016
Why Math Placement
2
Today’s Presenters
3
Assessing Remedial Assessments: How Useful are Placement Exams— and Can We Do Better? Judith Scott-Clayton Teachers College, Columbia University
Motivation: The Role, Prevalence, and Puzzle of College Remediation
5
No System is Perfect – Will always have mistakes in both directions
6
Our Research on Placement Validity (Scott-Clayton, 2012; Scott-Clayton et al. 2014)
7
Methodology
8
Methodology Figure 1 Classifications Based on Predicted Outcomes and Treatment Assignment Treatment assignment Assigned to remediation
Assigned to college-level
9
Predicted to Succeed in College-Level Course? No Yes (1) accurately placed (true positive)
(2) Under-placed (false positive)
(3) Over-placed (false negative)
(4) accurately placed (true negative)
Methodology Table 2. Predicted Severe Error Rates and Other Validity Metrics Using Alternative Measures for Remedial Assignment Test Scores A. LUCCS Sample Math Severe error rate Severe overplacement rate Severe underplacement rate CL success rate (>=C), if assigned to CL* Remediation rate English Severe error rate Severe overplacement rate Severe underplacement rate CL success rate (>=C), if assigned to CL* Remediation rate
10
Measures Used for Remedial Assignment HS GPA/ Test+HS Test HS GPA/ Test+HS Units Combined Scores Units Combined
COMPASS® Sample 23.9 5.3 18.5 67.5 76.1
N=37,813 22.9 5.0 17.9 69.8 74.7
21.4 4.7 16.7 72.4 74.7
-
-
-
33.4 4.5 28.9 71.6 80.5
N=34,697 29.4 2.2 27.2 81.8 79.8
29.3 2.7 26.6 81.4 79.8
-
-
-
Methodology Table 2. Predicted Severe Error Rates and Other Validity Metrics Using Alternative Measures for Remedial Assignment Test Scores A. LUCCS Sample Math Severe error rate Severe overplacement rate Severe underplacement rate CL success rate (>=C), if assigned to CL* Remediation rate English Severe error rate Severe overplacement rate Severe underplacement rate CL success rate (>=C), if assigned to CL* Remediation rate
11
Measures Used for Remedial Assignment HS GPA/ Test+HS Test HS GPA/ Test+HS Units Combined Scores Units Combined
COMPASS® Sample 23.9 5.3 18.5 67.5 76.1
N=37,813 22.9 5.0 17.9 69.8 74.7
21.4 4.7 16.7 72.4 74.7
-
-
-
33.4 4.5 28.9 71.6 80.5
N=34,697 29.4 2.2 27.2 81.8 79.8
29.3 2.7 26.6 81.4 79.8
-
-
-
Optimal cutoffs: trading off over/under placements
12
Summary of key findings
13
For more information: http://ccrc.tc.columbia.edu
[email protected]
CCRC is funded in part by: Alfred P. Sloan foundation, Bill & Melinda Gates Foundation, Lumina Foundation for Education, The Ford Foundation, National Science Foundation (NSF), Institute of Education Sciences of the U.S. Department of Education
14
The Opposing Forces that Shape Developmental Education Michelle Hodara Senior Researcher, Education Northwest Research Affiliate, Community College Research Center
System-wide Consistency vs. Institutional Autonomy
16
Developing consistent standards through consensus & evidence
17
Efficient vs. Effective Assessment
18
Supporting Progression vs. Upholding Standards
19
High-Quality Acceleration Models Maintain Pass Rates in College-Level Classes Student Performance by Track
73%
77%
59% 25%
Enroll in College-Level English
Earn a C or Better Among Students Who Enroll in CollegeLevel English
Non-accelerated Track
20
Accelerated Track
Questions to ponder…
21
Let Icarus Fly: The Potential for Multiple Measures Placement to Re-imagine Student Capacity in Mathematics John J. Hetts Senior Director of Data Science, CalPASS Plus/Educational Results Partnership Former Director of Institutional Research, Long Beach City College (In collaboration w/Peter Bahr, Loris Fagioli, Craig Hayward, Dan Lamoree, Mallory Newell, and Terrence Willett)
LBCC Multiple Measures Research
23
Alignment in Math Predicting Performance
Predicting Placement 1.00
24
.90 .80
.75
.70 .60 .50 .40 .30
.20
.20 .10
.00
.00
CST Math (z)
Last Math Grade
HSGPA
Logistic Regression Coefficients
Ordinal Regression Coefficients
1.00
.90
.73
.80 .70 .60 .50 .40 .30 .20
.20
.25
.10 .00
CST Math (z)
Last Math Grade
HSGPA
Re-imagined student capacity
25
Implementing Multiple Measures Placement: LBCC Transfer-level Math Placement Rates 35%
31% 30%
32% 29%
25%
F2011 LBUSD
20%
F2012 Promise Pathways Accuplacer Only
15%
F2012 Promise Pathways with Multiple Measures F2013 Pathways
10%
9%
9%
F2014 Pathways
5%
0%
Transfer Level Math
26
Comparison against traditional sequence: LBCC success rates in transfer-level courses 58% 56%
56%
55%
54% 52%
51%
50%
50%
49%
49%
48% 46% 44%
F2012 (p>.3)
F2013 (p = .06) Non-Pathways
27
Promise Pathways
F2014 (ns)
College-level course completion, other recent national examples: http://bit.ly/CCCSEMM
70% 60% 50%
Davidson County CC 2013-2015 65%
70%
68%
68%
48%
66% 64%
40%
62%
30%
60%
20%
59%
58%
10%
56%
0%
54%
Math Comparison
HS Data
Rules used for English and Math: HSGPA >=2.6 and completion of four years of mathematics including one year beyond Algebra 2 in HS
28
Ivy Tech 2014-2015
Math Accuplacer
HS Data
Rules used for English and Math: HSGPA >=2.6
Dramatic impacts on transfer Math completion within first two years – Long Beach City College 40%
36%
35% 30% 25%
21%
20% 15%
13%
26%
23%
21%
18%
12%
12%
10% 5%
4%
0%
F2011
F2012 Total
http://www.lbcc.edu/PromisePathways
29
Black
Hispanic
Asian
White
Multiple Measures Assessment Project
http://bit.ly/MMAP2015 30
http://bit.ly/MMAPRules
Projected impact on placement and success
40% 35%
31%
30% 25% 20% 15%
15%
10% 5% 0%
Math Historic (Placement)
31
Historic (Course-Taking)
Projected Success Rates Successful completion (C or better) of transfer-level course
45%
Placement into transfer-level 42%
70%
62%
62%
60% 50% 40% 30% 20% 10% 0%
Transfer-level Math Projected
Historic success rate
Projected success rate
Common Concerns/Multiple Measures Myths
32
Your test/system/school/segment is exceptionally unlikely to be different
33
Our test wasn’t different - Compass Course
Compass Test
Compass
HSGPA
HSGPA + Compass
Arithmetic
Pre-Algebra
.57
.34
.66
Algebra
Pre-Algebra
.36
.65
.80
Intermediate Algebra
Algebra
.47
.66
.84
College Algebra
Algebra
.41
.76
.88
College Algebra
College Algebra
.51
.76
.94
http://bit.ly/COMPASSValidation (Table 4 - Median Logistic R)
34
Our test wasn’t different - Accuplacer Math
Accuplacer
11th Grade GPA
Transfer - STEM
.19
.24
Transfer – Stats
.16
.31
Transfer – GEM
.09
.26
1 level below
.21
.28
2 levels below
.11
.26
3 levels below
.11
.23
4 levels below
.05
.19
MMAP (in preparation): Correlation with success (C or better) in course in CCC
35
Our tests weren’t different - NC
From Bostian (2016), North Carolina Waves GPA Wand, Students Magically College Ready adapted from research of Belfield & Crosta, 2012 – see also Table 1)
36
Our tests weren’t different - AK
From Hodara, M., & Cox, M. (2016), Developmental education and college readiness at the University of Alaska: http://bit.ly/HSGPAAK
37
Scant evidence that developmental education improves student outcomes
38
On balance, massive, costly semester-long intervention has far less impact than expected
http://bit.ly/CCRCDEVED
39
Even if students get lower grade in transfer-level course, potentially increases students’ likelihood of transfer
40
Students who get a C in transfer-level Math are more likely to transfer Transfer rates by level of first Math course and grade 80% 70%
67%
65%
63%
60% 50%
48%
48%
One-Level Below A
One-Level Below B
40% 30% 20% 10% 0% Transfer-Level A
Transfer-Level B
Transfer-Level C
Hayward & Fagioli (in preparation) Irvine Valley College Multiple Measures Research: First course enrolled in, Spring 2000 to Fall 2011 - transfer within 4 years of course
41
High School GPA is more predictive than tests for far longer than people think
42
HSGPA as good or better predictor for long time
MMAP (in preparation): correlations b/w predictor and success (C or better) in transfer-level course by # of semesters since HS
43
Utility of HSGPA vs. Compass for non-traditional students Traditional first-time students (