Improving the Targeting of Treatment: Emerging Research on Postsecondary Math Placement Policies

Improving the Targeting of Treatment: Emerging Research on Postsecondary Math Placement Policies Quantitative Leap! Webinar Series: Webinar 2 June 8,...
Author: Milo Marsh
1 downloads 0 Views 6MB Size
Improving the Targeting of Treatment: Emerging Research on Postsecondary Math Placement Policies Quantitative Leap! Webinar Series: Webinar 2

June 8, 2016

Why Math Placement

2

Today’s Presenters

3

Assessing Remedial Assessments: How Useful are Placement Exams— and Can We Do Better? Judith Scott-Clayton Teachers College, Columbia University

Motivation: The Role, Prevalence, and Puzzle of College Remediation

5

No System is Perfect – Will always have mistakes in both directions

6

Our Research on Placement Validity (Scott-Clayton, 2012; Scott-Clayton et al. 2014)

7

Methodology

8

Methodology Figure 1 Classifications Based on Predicted Outcomes and Treatment Assignment Treatment assignment Assigned to remediation

Assigned to college-level

9

Predicted to Succeed in College-Level Course? No Yes (1) accurately placed (true positive)

(2) Under-placed (false positive)

(3) Over-placed (false negative)

(4) accurately placed (true negative)

Methodology Table 2. Predicted Severe Error Rates and Other Validity Metrics Using Alternative Measures for Remedial Assignment Test Scores A. LUCCS Sample Math Severe error rate Severe overplacement rate Severe underplacement rate CL success rate (>=C), if assigned to CL* Remediation rate English Severe error rate Severe overplacement rate Severe underplacement rate CL success rate (>=C), if assigned to CL* Remediation rate

10

Measures Used for Remedial Assignment HS GPA/ Test+HS Test HS GPA/ Test+HS Units Combined Scores Units Combined

COMPASS® Sample 23.9 5.3 18.5 67.5 76.1

N=37,813 22.9 5.0 17.9 69.8 74.7

21.4 4.7 16.7 72.4 74.7

-

-

-

33.4 4.5 28.9 71.6 80.5

N=34,697 29.4 2.2 27.2 81.8 79.8

29.3 2.7 26.6 81.4 79.8

-

-

-

Methodology Table 2. Predicted Severe Error Rates and Other Validity Metrics Using Alternative Measures for Remedial Assignment Test Scores A. LUCCS Sample Math Severe error rate Severe overplacement rate Severe underplacement rate CL success rate (>=C), if assigned to CL* Remediation rate English Severe error rate Severe overplacement rate Severe underplacement rate CL success rate (>=C), if assigned to CL* Remediation rate

11

Measures Used for Remedial Assignment HS GPA/ Test+HS Test HS GPA/ Test+HS Units Combined Scores Units Combined

COMPASS® Sample 23.9 5.3 18.5 67.5 76.1

N=37,813 22.9 5.0 17.9 69.8 74.7

21.4 4.7 16.7 72.4 74.7

-

-

-

33.4 4.5 28.9 71.6 80.5

N=34,697 29.4 2.2 27.2 81.8 79.8

29.3 2.7 26.6 81.4 79.8

-

-

-

Optimal cutoffs: trading off over/under placements

12

Summary of key findings

13

For more information: http://ccrc.tc.columbia.edu

[email protected]

CCRC is funded in part by: Alfred P. Sloan foundation, Bill & Melinda Gates Foundation, Lumina Foundation for Education, The Ford Foundation, National Science Foundation (NSF), Institute of Education Sciences of the U.S. Department of Education

14

The Opposing Forces that Shape Developmental Education Michelle Hodara Senior Researcher, Education Northwest Research Affiliate, Community College Research Center

System-wide Consistency vs. Institutional Autonomy

16

Developing consistent standards through consensus & evidence

17

Efficient vs. Effective Assessment

18

Supporting Progression vs. Upholding Standards

19

High-Quality Acceleration Models Maintain Pass Rates in College-Level Classes Student Performance by Track

73%

77%

59% 25%

Enroll in College-Level English

Earn a C or Better Among Students Who Enroll in CollegeLevel English

Non-accelerated Track

20

Accelerated Track

Questions to ponder…

21

Let Icarus Fly: The Potential for Multiple Measures Placement to Re-imagine Student Capacity in Mathematics John J. Hetts Senior Director of Data Science, CalPASS Plus/Educational Results Partnership Former Director of Institutional Research, Long Beach City College (In collaboration w/Peter Bahr, Loris Fagioli, Craig Hayward, Dan Lamoree, Mallory Newell, and Terrence Willett)

LBCC Multiple Measures Research

23

Alignment in Math Predicting Performance

Predicting Placement 1.00

24

.90 .80

.75

.70 .60 .50 .40 .30

.20

.20 .10

.00

.00

CST Math (z)

Last Math Grade

HSGPA

Logistic Regression Coefficients

Ordinal Regression Coefficients

1.00

.90

.73

.80 .70 .60 .50 .40 .30 .20

.20

.25

.10 .00

CST Math (z)

Last Math Grade

HSGPA

Re-imagined student capacity

25

Implementing Multiple Measures Placement: LBCC Transfer-level Math Placement Rates 35%

31% 30%

32% 29%

25%

F2011 LBUSD

20%

F2012 Promise Pathways Accuplacer Only

15%

F2012 Promise Pathways with Multiple Measures F2013 Pathways

10%

9%

9%

F2014 Pathways

5%

0%

Transfer Level Math

26

Comparison against traditional sequence: LBCC success rates in transfer-level courses 58% 56%

56%

55%

54% 52%

51%

50%

50%

49%

49%

48% 46% 44%

F2012 (p>.3)

F2013 (p = .06) Non-Pathways

27

Promise Pathways

F2014 (ns)

College-level course completion, other recent national examples: http://bit.ly/CCCSEMM

70% 60% 50%

Davidson County CC 2013-2015 65%

70%

68%

68%

48%

66% 64%

40%

62%

30%

60%

20%

59%

58%

10%

56%

0%

54%

Math Comparison

HS Data

Rules used for English and Math: HSGPA >=2.6 and completion of four years of mathematics including one year beyond Algebra 2 in HS

28

Ivy Tech 2014-2015

Math Accuplacer

HS Data

Rules used for English and Math: HSGPA >=2.6

Dramatic impacts on transfer Math completion within first two years – Long Beach City College 40%

36%

35% 30% 25%

21%

20% 15%

13%

26%

23%

21%

18%

12%

12%

10% 5%

4%

0%

F2011

F2012 Total

http://www.lbcc.edu/PromisePathways

29

Black

Hispanic

Asian

White

Multiple Measures Assessment Project

http://bit.ly/MMAP2015 30

http://bit.ly/MMAPRules

Projected impact on placement and success

40% 35%

31%

30% 25% 20% 15%

15%

10% 5% 0%

Math Historic (Placement)

31

Historic (Course-Taking)

Projected Success Rates Successful completion (C or better) of transfer-level course

45%

Placement into transfer-level 42%

70%

62%

62%

60% 50% 40% 30% 20% 10% 0%

Transfer-level Math Projected

Historic success rate

Projected success rate

Common Concerns/Multiple Measures Myths

32

Your test/system/school/segment is exceptionally unlikely to be different

33

Our test wasn’t different - Compass Course

Compass Test

Compass

HSGPA

HSGPA + Compass

Arithmetic

Pre-Algebra

.57

.34

.66

Algebra

Pre-Algebra

.36

.65

.80

Intermediate Algebra

Algebra

.47

.66

.84

College Algebra

Algebra

.41

.76

.88

College Algebra

College Algebra

.51

.76

.94

http://bit.ly/COMPASSValidation (Table 4 - Median Logistic R)

34

Our test wasn’t different - Accuplacer Math

Accuplacer

11th Grade GPA

Transfer - STEM

.19

.24

Transfer – Stats

.16

.31

Transfer – GEM

.09

.26

1 level below

.21

.28

2 levels below

.11

.26

3 levels below

.11

.23

4 levels below

.05

.19

MMAP (in preparation): Correlation with success (C or better) in course in CCC

35

Our tests weren’t different - NC

From Bostian (2016), North Carolina Waves GPA Wand, Students Magically College Ready adapted from research of Belfield & Crosta, 2012 – see also Table 1)

36

Our tests weren’t different - AK

From Hodara, M., & Cox, M. (2016), Developmental education and college readiness at the University of Alaska: http://bit.ly/HSGPAAK

37

Scant evidence that developmental education improves student outcomes

38

On balance, massive, costly semester-long intervention has far less impact than expected

http://bit.ly/CCRCDEVED

39

Even if students get lower grade in transfer-level course, potentially increases students’ likelihood of transfer

40

Students who get a C in transfer-level Math are more likely to transfer Transfer rates by level of first Math course and grade 80% 70%

67%

65%

63%

60% 50%

48%

48%

One-Level Below A

One-Level Below B

40% 30% 20% 10% 0% Transfer-Level A

Transfer-Level B

Transfer-Level C

Hayward & Fagioli (in preparation) Irvine Valley College Multiple Measures Research: First course enrolled in, Spring 2000 to Fall 2011 - transfer within 4 years of course

41

High School GPA is more predictive than tests for far longer than people think

42

HSGPA as good or better predictor for long time

MMAP (in preparation): correlations b/w predictor and success (C or better) in transfer-level course by # of semesters since HS

43

Utility of HSGPA vs. Compass for non-traditional students Traditional first-time students (

Suggest Documents