Ensuring Quality in Undergraduate STEM Programs: New Frameworks for Transforming STEM Teaching and Learning AAC&U Annual Meeting January 22-25, 2014 Washington, DC Presenters: Linda Slakey, AAC&U and Association of American Universities (AAU) Susan Elrod, Fresno State and AAC&U/Project Kaleidoscope Emily Miller, AAU Geoff Chase, San Diego State University Marco Molinaro, UC Davis Adrianna Kezar, University of Southern California

Session Outcomes ¤  Learn about two national research-based STEM Education Framework projects ¤  Better understand the elements required for comprehensive undergraduate STEM reform ¤  Understand the benefits and challenges involved in mounting institution-wide STEM reform initiatives ¤  Gain practical knowledge about implementation of framework elements from project leaders ¤  Apply framework elements to your campus

Frameworks as tools for change ¤  Why are we talking about frameworks? Creating a new normal – instruction that aligns with what we know about how students learn – requires change across the institution. ¤  Why are we focused on STEM in these initiatives? §  STEM faculty lag other disciplines in adopting studentcentered pedagogy. §  Economic concerns have created a sense of national urgency.

¤  STEM subjects are part of the liberal arts!

Keck/PKAL Framework Project This initiative aims to develop a comprehensive, institutional STEM Education Effectiveness Framework that will help campus leaders translate national recommendations and research for improving student learning and success in STEM into scalable and sustainable actions on particular campuses.

What a Framework Helps Accomplish ¤  Articulates a vision or direction for change

¤  Serves as accountability tool for maintaining momentum on the change

¤  Audits or evaluates where you are now and where you want to go

¤  Serves as catalyst for priority setting

¤  Charts a path for reaching the vision and goals

¤  Ensures collective leadership

¤  Creates a common language and vision around the change ¤  Pinpoints needed interventions and strategies ¤  Fosters learning

¤  Justifies resources (financial and human) for the vision ¤  Maintains focus and momentum over the long time period change often takes

Participating Campuses

PI: Susan Elrod; Evaluator: Adrianna Kezar

INITIAL Framework 1.0 Vision

Campus Teams: Faculty Institutional Leaders Office of Institutional Research

Align w/ Institutional Priorities

Landscape Analysis

Identify & Analyze Challenges Choose Strategies

Implement Measure & Disseminate Results

Keck/PKAL STEM Education Framework v 2.3 Informed by research, reports and experience of project teams Mapped to different levels of challenge/action (Department, College, GE/Core, Central Admin)

Vision Align w/ Institutional Priorities

Landscape Example Vision: Improved STEM graduation Types of Landscape Analyses: Enrollment #s Retention % Remedial % Unit completion Course data Progression in series Graduation rates Learning gains Engagement (NSSE) Participation Etc.

Leadership

Identify & Analyze Challenges

Examples: •  Learning/ Depth/ Breadth •  High levels of remedial math among URM students •  Poor URM student retention •  Lack of faculty expertise in pedagogy

Choose Strategies

Determine Readiness

Measure Results

Implement

Decision Point

Examples: •  Studio Classes •  Peer learning •  STEM FYE •  Summer Bridge Program •  Advising •  Faculty Development

Receptivity

??? about: •  Faculty Leaders •  Buy-In •  Workload •  IR Support •  Institutional Commitment •  Resources •  Sustainability

Descriptions of:

Toolkit for:

What/Who will be/ is required at different institutional levels for action: plan, pilot, assess, repeat, scale.

How success is measured, processes for continuous improvement

Disseminate Results and Plan Next Steps

Readiness

Action

Success

AAU Undergraduate STEM Education Initiative ¤  Objective: Influence the culture of STEM departments at AAU universities so that faculty are encouraged and supported to use teaching practices proven by research to be more effective in engaging students in STEM education and in helping students learn.

Goals "AAU is not conducting another study or research project on STEM education. We are moving to implement the results of the latest research into science and math pedagogy.” Hunter Rawlings, AAU President

¤  Framework ¤  Project Sites ¤  Institutional Levers: Recognition, Incentives & Rewards ¤  Federal Levers: Promotion, Recognition & Rewards ¤  Promising Practices

Framework for Systemic Change in Undergraduate STEM Teaching and Learning The Framework outline provides a set of key institutional elements that need to be addressed in order to bring about sustainable change. • 

Faculty Members

• 

Department Chairs

• 

College & University Administrators

• 

Institutions

Three Layers Pedagogical Practices ¤  Articulated Learning Goals ¤  Educational Practices ¤  Assessment ¤  Access

Scaffolding

Cultural Change

¤  Provide Faculty Professional Development

¤  Leadership Commitment

¤  Provide Faculty with Accessible Resources ¤  Collect Data on Program Performance ¤  Align Future Facilities Planning

¤  Establish Strong Measures of Teaching Excellence ¤  Align Incentives with the Expectation of Teaching Excellence

AAU STEM Project Sites

San Diego State University ¤  Project Vision & Goals ¤  Measures of Success ¤  Interventions ¤  Project Progress ¤  Lessons Learned Special thanks: Ed Nuhfer and co-developers of SLCI Student and instructors of SDSU GE science courses

San Diego State University ¤  Vision and Goal: Strengthen our institutional effectiveness in developing science literacy through our lower- and upper-division GE courses ¤  Measures of Success and How We Are Doing On Them: ¤  1. Gains in science literacy as measured by the Science Literacy Concept Inventory (SLCI; Nuhfer et al.) ¤  Progress: Baseline results from Spring 2013 SLCI deployment (focus) ¤  2. Mapping of SLCI items onto GE program-level outcomes ¤  Progress: Mapping of SLCI items onto multiple and different GE program-level outcomes, largely attributed to the outcomes’ metadisciplinary nature . . . and that’s OK! ¤  3. Increased articulation of course-level activities with GE program-level outcomes through the syllabus, materials, and lectures/discussions . . . and modeling the way! ¤  Progress: Awareness being raised through working-groups and discussions

SLCI Example Question Q06: To help us to understand the lunar phases, we have set up a basketball, a baseball, and a golf ball to represent respectively the Sun, Earth and the moon. What method of science are we employing? A. Experiment. Moving the balls can allow us to measure the size of the shadow that one ball casts on another ball. B. Modeling. Moving the balls helps us to perceive the positions of the celestial bodies that might explain the observed phases. C. Multiple working hypotheses. Moving the balls can allow us to determine whether the lunar phases were different during the ice ages. D. None. Moving the balls differs from reality to such an extent that it is an ineffective way to understand lunar phases.

¤  Associated SLCI responses with student information from SIMS

% of Students

¤  Deployed Science Literacy Concept Inventory (SLCI) across seven lower-division and six upper-division GE sections

n = 1,903 of 2,794 (39%) Mean = 66 ± 19 Median = 72

SLCI Score

Student Information (Independent Variables) Demographics

Preparedness & Performance

SDSU “Treatments”

Age

SAT (ACT) Score

Student Level

Gender

GPA

Upper- vs. Lower-Division GE

Ethnicity, Race

Units Completed

First Generation

Course Modality

English as 1st Language

Major (Sci., Eng., Other)

Service Area

Residence Hall

So What Influences the SLCI Score of a Student?

¤

Demographics

Preparedness & Performance

SDSU “Treatments”

Age

SAT (ACT) Score

Student Level

Gender

GPA

Foundations, Exploration

Ethnicity, Race

Total Units Completed

First Generation

Course Modality

English as 1st Language

Major (Sci., Eng., Other)

Service Area

Residence Hall

¤

¤

San Diego State University ¤  Intervention: Use SLCI to raise awareness of GE outcomes in course-focused faculty ¤  Progress: Evolving to student-gains-based and course-treatment-based use of SLCI ¤  Lesson: Progress favored by leadership, grass-roots buy-in, incentives, and resources

UC DAVIS ¤  VISION: STEM Instructional Decisions/Approaches are based on research and data and guide improvements that increase Student Success ¤  Goals ¤  Bring together institutional data relevant to student progress and outcomes ¤  Make instructional data accessible and easy to use ¤  Gauge instructional practices over time while fostering an evidence-based instructional culture ¤  Encourage clear learning outcomes and assessment measures ¤  Catalyze and nurture instructional experiments

Increasing Student Success Receptive/ Curious

Vision Align w/ Institutional Priorities

Landscape (readiness) Example Vision: Increased student success Types of Landscape Analyses: Enrollment #s Retention % Remedial % Unit completion Course completion data Progression in series Graduation rates Learning gains Teaching Practices Department culture/ purpose Majors vs service

Identify & Analyze Challenges

Receptive/ Curious

Choose Strategies

Determine Readiness

Measure Results

Implement

Decision Point

Examples:

Examples:

??? about:

Descriptions of: Toolkit for:

Learning/ Depth/Breadth

Flipped/Adap. Learning/ POGILS

Faculty/Admin Leaders

BIO instructional Implement change various tools

Buy-In

Pre/Post bio, chem, math, phys

Placement practices

Grad student involvement

Poor URM student retention

Course LOs

Lack of faculty expertise in pedagogy

Faculty Development

Advising

Pre/Post learning/ attitudes

Workload Expt Support

Share results between departments

Observations of teaching Departmental & practice Faculty interest Motivate and Sustainability measure change

Three Layers UC Davis Focus Pedagogical Practices

Scaffolding

¤  Provide Faculty Professional ¤  Articulated Learning Development Goals ¤  Educational Practices ¤  Assessment ¤  Access

¤  Provide Faculty with Accessible Resources ¤  Collect Data on Program Performance ¤  Align Future Facilities Planning

Cultural Change ¤  Leadership Commitment ¤  Establish Strong Measures of Teaching Excellence ¤  Align Incentives with the Expectation of Teaching Excellence

LANDSCAPE STEM VS Non-STEM Losses Landscape (readiness) Percentage Loss of Students from Graduation by Term (Fall 2006 entering class) 70% Loss in Yr1

Yr2

Yr3

Yr4

60%

Yr5

58%

57%

Yr6

59% 59%

59% 58% 58%

58%

STEM, STEM, URG, FG

54% 51%

STEM, STEM, RG, ALL

44% 38%

40% 32%

30%

28% 24% 23%

23% 20% 20% 19%

20%

15% 18%

10%

29% 27% 25%

27%

34% 33%

22% 21%

15% 14%

16% 14% 15%

43%

37% 37% 37%

37%

STEM, STEM, RG, NOT FG

noSTEM, noSTEM, URG, ALL 25%

29% 28% 28%

28% 27%

29% 28% 28%

28%

noSTEM, noSTEM, URG, FG noSTEM, noSTEM, URG, NOT FG

22% 21% 21%

20% 19%

22% 22% 22%

22%

17%

noSTEM, noSTEM, RG, ALL noSTEM, noSTEM, RG, FG noSTEM, noSTEM, RG, NOT FG

12%

ALL STUDENTS Oct-13

Jul-13

Apr-13

Jan-13

Oct-12

Jul-12

Apr-12

Jan-12

Oct-11

Jul-11

Apr-11

Jan-11

Oct-10

Jul-10

Apr-10

Jan-10

Jul-09

Oct-09

Apr-09

Jan-09

Oct-08

Jul-08

Apr-08

Jan-08

Oct-07

STEM, STEM, RG, FG

31%

29% 28%

24% 22% 22%

36% 36% 36%

43% 42% 43%

4%

Jul-07

Apr-07

Jan-07

0%

4% 3% 1% 0% 4% 1% 0% 1% 0%

34%

32%

38%

37% 35%

18%

10% 9% 14% 13% 7% 10% 8%

43% 42% 42%

42% 40%

42%

Oct-06

Percentage Loss

STEM, STEM, URG, NOT FG

49% 47%

50%

STEM, STEM, URG, ALL

LANDSCAPE: TARGETS and OPPORTUNITIES ¤  BIOLOGY ¤  Willing Assoc. Dean – changed intro labs in 2008 ¤  Gates grant opportunity

Receptive/ Curious?

¤  Interested faculty and experimental TAs

¤  CHEMISTRY ¤  Placement test difficulties ¤  Willing chair and many faculty (many lecturers involved), education interested graduate students ¤  Large service system

¤  MATH, PHYSICS, ENGINEERING … similar

LANDSCAPE PATHS: Major in/Major out Landscape (readiness)

Fall 2006 Student Flows

Intro Biology Example Implement

Highly Structured Course

Adaptive Learning Modules [Carnegie Mellon OLI] Flipped Classroom [UW Bio Group] In-class group exercises Practice exams, Reading quizzes EXPECTED (hoped for) reduction in achievement gap

~45%

PRE/POST Example: Content Implement

Measure Results

BIOLOGY (Preliminary results) average # of questions answered correctly on Pre-Post :  Total students Before:  13/26 After:  17/26

4 point (15% gain)

Top Quartile students Before:  18/26 After:  19/26 Bottom Quartile students Before:  8/26 4 point (30% gain) After:  15/26

PRE/POST Example: Attitudes Implement

Measure ! Results !

Class!A! Pre! 63.4!

Post! 60.9!

Class!B! Change! 62.5!

Pre! 63.4!

Post! 58.9!

Change! 64.5!

Time!X!Group! Interaction!! p6Value! .075!

Overall! Real%World% 71.4% 70.6% 60.8% 73.7% 68.0% 65.7% .010*% Connections% Problem6 Solving! 49.1! 47.1! 62.0! 48.5! 45.6! 62.9! .618! Difficulty! Enjoyment% 59.6% 59.9% 0.3% 60.2% 56.1% 64.1% .023*% Problem6 64.2% 64.0% 60.2% 66.1% 61.2% 64.9% .012*% Solving%Effort% Conceptual! 64.0! 65.5! Connections/! 68.3! 65.0! 63.3! 69.5! .203! Memorization! Problem6 Solving% 67.0% 69.6% 2.6% 68.1% 65.9% 62.2% .042*% Strategies% Reasoning! 76.8! 73.5! 63.3! 78.3! 73.1! 65.2! .384! ! Note: A statistically significant Time X Group Interaction indicates that the amount Class A and Class B changed from pre to post was significantly different.

CLASS BIO instrument from Boulder

NEXT STEPS ¤  BIOLOGY ¤  Second Experiment Spring 2014, multiple Summer 2014, full flip Fall 2014 ¤  Expansion to 2B and 2C courses ¤  Full Pre/Post and classroom observations for 2A, B, C courses

¤  CHEMISTRY ¤  Pre-pilot with POGIL activities in discussion ¤  Full pilot of POGIL in 1 or more 2A sections ¤  Pre/Post content and attitudes for all 2A courses ¤  Defining course learning objectives for first 2 years ¤  Developing and administering Post/Post for first 2 years and classroom observations

¤  MATH, PHYSICS, ENGINEERING similar with customization

Table Discussions Goals: ¤  Encourage and support faculty to use student-centered, evidence-based, active learning pedagogy in undergraduate STEM courses ¤  Improve student engagement, learning, and achievement in STEM education

How might these frameworks help your campus achieve these goals?