GAUGING MOOC LEARNERS’ ADHERENCE TO THE DESIGNED LEARNING PATH DAN DAVIS, GUANLIANG CHEN, CLAUDIA HAUFF, GEERT-JAN HOUBEN
TU DELFT WEB INFORMATION SYSTEMS GROUP
WHAT IS A LEARNING PATH ?
WHAT IS A LEARNING PATH ? THE SEQUENCE OF EVENTS A STUDENT TAKES TOWARDS A LEARNING OBJECTIVE
RQ
to what extent do learners adhere to the designed learning path?
VIDEO
QUIZ
WATCH START SUBMIT END
PROGRESS FORUM VIEW
START SUBMIT END
DESIGNED LEARNING PATHS Data Analysis Data Analysis
Functional Programming Programming Functional
Framing Framing
ResponsibleInnovation Innovation Responsible
QUIZ QUIZ VIDEO START SUBMIT v
Q
v
Q
Q
v
Q
Q
v
v
Q
Q
Q
v
v
Q
Q
v
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
v
F
F
F
v
F
F
F
v
QUIZ END Q
v
v
F
F
F
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
v
F
F
F
v
F
F
F
v
F
F
v
v
v
v
v
F
F
F
F
v
QUIZ QUIZ START SUBMIT
FORUM START SUBMIT END
VIDEO v
Q
FORUM START SUBMIT END
VIDEO v
Q
QUIZ QUIZ START SUBMIT
VIDEO v
Q
QUIZ END
v
Q
Q
Q
QUIZ END Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
F
F
F
RELATED WORK Demographic differences in how students navigate through MOOCs. In L@S ‘14 PHILIP GUO & KATHARINA REINECKE
Identifying latent study habits by mining learner behavior patterns in massive open online courses. In CIKM ’14 MIAOMIAO WEN & CAROLYN ROSÉ
* PASS CHAINS QUIZ TRIES VIDEOS RATE PASS/NON QUESTIONS ACCESSED
MOOC
ENROLLED
FP101x
37k
5.3%
1.06M/807K
288
1
67.5%
RI101x
9k
4.3%
66k/30k
75
1—3
49.7%
Frame101x
34k
2.4%
95k/141k
26
2
51%
EX101x
33k
6.5%
1.02M/855k
136
2
45%
* Guo, P. & Reinecke, K. Demographic differences in how students navigate through MOOCs (2014) : 78%
* PASS CHAINS QUIZ TRIES VIDEOS RATE PASS/NON QUESTIONS ACCESSED
MOOC
ENROLLED
FP101x
37k
5.3%
1.06M/807K
288
1
67.5%
RI101x
9k
4.3%
66k/30k
75
1—3
49.7%
Frame101x
34k
2.4%
95k/141k
26
2
51%
EX101x
33k
6.5%
1.02M/855k
136
2
45%
* Guo, P. & Reinecke, K. Demographic differences in how students navigate through MOOCs (2014) : 78%
FORUM START
3
VIDEO
4
FORUM START
1
QUIZ START
VIDEO
1
QUIZ START
FORUM SUBMIT
6
3
1
QUIZ SUBMIT QUIZ END
QUIZ SUBMIT FORUM END
3
6
EX101x
QUIZ END
PROGRESS
PROGRESS
FORUM START
FORUM START
1
7
VIDEO
5
QUIZ START
VIDEO
1
1
QUIZ START
FORUM SUBMIT
FORUM SUBMIT
1
7
1
QUIZ SUBMIT
QUIZ SUBMIT QUIZ END
Frame101x
FORUM END
12
1
FP101x
7 3
FORUM SUBMIT
6 PROGRESS
FORUM END
QUIZ END
1
RI101x
10
1 PROGRESS
FORUM END
FORUM START
3
VIDEO
4
FORUM START
1
QUIZ START
VIDEO
1
QUIZ START
FORUM SUBMIT
6
3
1
EACH COURSE IS UNIQUE QUIZ SUBMIT
QUIZ END
6
QUIZ SUBMIT
FORUM END
3
EX101x
QUIZ END
PROGRESS
PROGRESS
FORUM START
FORUM START
1
7
VIDEO
5
QUIZ START
VIDEO
1
1
QUIZ START
FORUM SUBMIT
FORUM SUBMIT
1
7
1
QUIZ SUBMIT
QUIZ SUBMIT QUIZ END
Frame101x
FORUM END
12
1
FP101x
7 3
FORUM SUBMIT
6 PROGRESS
FORUM END
QUIZ END
1
RI101x
10
1 PROGRESS
FORUM END
FORUM START
3
VIDEO
4
FORUM START
1
QUIZ START
VIDEO
1
QUIZ START
FORUM SUBMIT
6
3
1
EACH COURSE IS UNIQUE QUIZ SUBMIT
QUIZ END
6
QUIZ SUBMIT
FORUM END
3
EX101x
3
QUIZ END
PROGRESS
PROGRESS
FORUM START
FORUM START
ANALYZE ACCORDINGLY 1
7
5
QUIZ START
VIDEO
1
1
QUIZ START
FORUM SUBMIT
FORUM SUBMIT
1
7
1
QUIZ SUBMIT
QUIZ SUBMIT QUIZ END
Frame101x
FORUM END
12
1
FP101x
7 VIDEO
FORUM SUBMIT
6 PROGRESS
FORUM END
QUIZ END
1
RI101x
10
1 PROGRESS
FORUM END
APPROACH 1. VIDEO INTERACTIONS 2. BEHAVIOR PATTERN CHAINS 3. EVENT TYPE TRANSITIONS
FINDINGS
1. VIDEO INTERACTIONS DO LEARNERS WATCH VIDEO LECTURES IN THE PRESCRIBED ORDER?
VIDEO INTERACTIONS Executed Paths
Passing Week 1
Week 2
Non-Passing
FP101x Designed Lecture Order
Week 3
VIDEO INTERACTIONS Passing Week 1
Passing Week 2
Week 3
Week 1
Non-Passing
Non-Passing
FP101x
Frame101x Passing
Passing Week 2
Week 3
Week 3 Week 1
Week 1
Non-Passing
RI101x
Week 2
Week 3
Week 2
Non-Passing
EX101x
2. BEHAVIOR PATTERN CHAINS WHAT ARE THE MOST COMMON SEQUENCES IN THE EXECUTED LEARNING PATHS?
EXAMPLE
EIGHT-STEP CHAIN FORUMEND
LECTUREWATCH QUIZEND
QUIZSTART
PROGRESS
QUIZSUBMIT
LECTUREWATCH
QUIZSUBMIT
* PASS CHAINS QUIZ TRIES VIDEOS RATE PASS/NON QUESTIONS ACCESSED
MOOC
ENROLLED
FP101x
37k
5.3%
1.06M/807K
288
1
67.5%
RI101x
9k
4.3%
66k/30k
75
1—3
49.7%
Frame101x
34k
2.4%
95k/141k
26
2
51%
EX101x
33k
6.5%
1.02M/855k
136
2
45%
* Guo, P. & Reinecke, K. Demographic differences in how students navigate through MOOCs (2014) : 78%
OPEN CARD SORTING
MOTIF
A RECURRING OR REPEATED ELEMENT
FP101x
FREQ TOTAL
FREQ PASS
FREQ FAIL
552,363 (29.4%)
328,995 (30.8%)
223,368 (27.7%)
149,784 (8%)
59,498 (5.6%)
90,286 (11.2%)
100,179 events w/ >1 X = SUBMIT (5.3%)
50,415 (4.7%)
49,764 (6.2%)
67,722 (6.3%)
32,106 (4%)
MOTIF 1. QUIZ COMPLETE XQUIZ events only with at least 1 X = SUBMIT
2. BINGE WATCHING WATCH events only
3. LECTURE -> QUIZ COMPLETE WATCH event(s) followed by XQUIZ
4. QUIZ COMPLETE -> FORUM XQUIZ events w/ >1 X = SUBMIT followed by XFORUM events
99,828 (5.3%)
3. EVENT TYPE TRANSITIONS HOW DO STUDENTS NAVIGATE BETWEEN CERTAIN EVENT TYPES?
EVENT TYPE TRANSITIONS (EXECUTED)
Frame101x Non-Passing Markov Model State Visualization
EVENT TYPE TRANSITIONS (EXECUTED)
Frame101x Non-Passing
Frame101x Passing
EVENT TYPE TRANSITIONS (EXECUTED)
FORUM START
FORUM START
QUIZ START
VIDEO
QUIZ END
35%
Frame101x Non-Passing
VIDEO
FORUM SUBMIT
FORUM END
QUIZ SUBMIT
PROGRESS
32%
QUIZ START
QUIZ END
44%
Frame101x Passing
FORUM SUBMIT
QUIZ SUBMIT
FORUM END
21%
28% PROGRESS
44%
EXAMPLE
FEEDBACK DAVIS, D., CHEN, G., JIVET, I., HAUFF, C., & HOUBEN, G. J. (2016). ENCOURAGING METACOGNITION & SELF-REGULATION IN MOOCS THROUGH INCREASED LEARNER FEEDBACK. LAK 2016 LAL WORKSHOP
Frame101x Non-Passing
RI101x Non-Passing
Frame101x Passing
EX101x Non-Passing
RI101x Passing
FP101x Non-Passing
EX101x Passing
FP101x Passing
APPLICATIONS
1. BEYOND PASS-FAIL SEGMENTATION 2. PATTERN MINING & CONFORMANCE CHECKING 3. POSITIVE DEVIANCE
BIT.LY/WIS-LEARNING-ANALYTICS
THANK YOU
DAN DAVIS, GUANLIANG CHEN, CLAUDIA HAUFF, GEERT-JAN HOUBEN
TU DELFT WEB INFORMATION SYSTEMS GROUP
