Relate PennDOT’s strategic plan for LTAP evaluation Provide current forms used for all levels of Training and Technical Assistance evaluation Facilitate an exchange of ideas to enhance your (and PennDOT’s) evaluation program
2
2008 PennDOT Strategic Plan for LTAP Engaged a Strategic Planning Committee, LTAP Advisory Committee, and LTAP customers— road masters and public works directors Established Evaluation as a “Priority Goal”
3
Goal 4: Identify the value of LTAP to local governments through performance measures.
Performance Indicators: Establishment of quantifiable measures of customer value through compiled data from pre/post workshop quizzes and evaluations Measurement of customer value through compiled Kirkpatrick Level 3 and Level 4 evaluations (ongoing) 4
LTAP Comprehensive Evaluation Report • Historical analysis of training and technical assistance
offerings • Comparison of data generated with strategic goal
statement • Assessment of processes and forms used for evaluation • Consideration of evaluation framework
Report provides recommendations for refining the overall evaluation process and employing simplified data collection forms. 5
Structure all evaluations on Kirkpatrick levels Increase return rates Quantify value!
6
Training Evaluations Level 1 • 3,123 attendees • 2,601 completed evaluations • Maintenance courses – return rate of 86% • Safety courses – return rate of 80% • Scores were consistently high
7
The training materials were relevant and easy to read. Low 1
High 2
3
4
5
8
Minor updates to Level 1 form
1 9
Observation: Evaluation form return rates were lower than quiz form return rates. Discussion: What has worked for you to ensure high return rates?
10
Training Evaluations Level 2 • Pre- and post-course quizzes since June 2008 − Six items measure pre-course knowledge − Six items measure post-course knowledge − Six additional items measure post-course knowledge for Roads Scholars • For all participants, report compared pre- and
post-test scores for statistical significance
11
Training Evaluations Level 2 (cont.) • For Roads Scholars, report also compared
post-test average to average of Roads Scholar-specific items Recommendation: Update Roads Scholar questions where participants scored the same or better than general post-test average. 12
Course Title
Engineering and Traffic Studies Risk Management and Tort Liability Roadway Safety Improvement Program Traffic Calming Traffic Signs Work Zone Traffic Control
Significant?
Post-Course Mean for Roads Scholars
Roads Scholars Questions Only Mean
Difference
Significant?
+.50
yes
5.00
5.67
+.67
no
5.84
+.21
no
5.17
5.33
+.16
no
3.78
2.33
‐1.45
yes
2.30
5.00
+2.70
yes
5
3.71
5.52
+1.81
yes
5.60
5.60
+.00
no
66
44
3.82
4.82
+1.00
yes
4.82
5.57
+.75
yes
58
15
2.57
4.05
+1.48
yes
4.87
4.80
‐.07
no
Number of Participants
Number of Roads Scholars
Pre-Course Mean
20
3
4.90
5.40
32
6
5.63
9
6
21
Post-Course Difference Mean
13
Discussion: Should Roads Scholar quiz questions be . . . A. Easier than post-course questions? B. The same difficulty level as post-course questions? C. More difficult than post-course questions?
14
For PennDOT’s program, the answer was C.
How could you make the Roads Scholar questions more difficult, but fair?
7. When performing a roadway condition survey, which of the following distress characteristics need to be accounted for: a. Type of distress b. Severity of the distress c. Extent of the distress
7. The most appropriate treatment for the condition in this picture is: a. Crack seal b. Bituminous overlay c. Base repair d. Seal coat
d. All of the above
17
8. Placing a chip seal on an asphalt road is an example of:
8. Delaying the oxidation of asphalt pavement is accomplished through:
a. Routine maintenance
a. Routine maintenance
b. Preventive maintenance
b. Preventive maintenance
c. Deferred maintenance
d. Rehabilitation
c. Deferred maintenance
d. Rehabilitation
18
10. Which of the following elements of a successful pavement preservation program can be eliminated? a. Determining the correct time to intervene.
10. In managing pavement deterioration, the most important factor in determining the cost of a treatment is: a. The cause of the distress
b. Determining the cause of the distress
b. The traffic pattern of the roadway
c. Identifying and applying the correct treatment
c. The timing of the treatment
d. None of the above
d. The type of treatment used 19
Update Questions
PennDOT Review
Task Training ‐ Level 1 Revise Level 1 Evaluation PennDOT Review Complete Evaluation Implemented Training ‐ Level 2 Revise Roads Scholar Test Questions Road Surface Management PennDOT Review Complete Questions Complete Roadway Safety Improvement Program PennDOT Review Complete Questions Complete Posting & Bonding of Local Roads PennDOT Review Complete Questions Complete Work Zone Traffic Control Asphalt Roads Common Maintenance Problems PennDOT Review Complete Questions Complete Risk Management/Tort Liability PennDOT Review Complete Questions Complete Traffic Signs Roadside Vegetation Control Unpaved & Gravel Roads Common Maintenance Practices Engineering & Traffic Studies PennDOT Review Complete Q i C l
Implement New Questions
Finalize Questions
May‐09
Jun‐09
Jul‐09
Aug‐09
Sep‐09
Oct‐09
Evaluate for Statistical Significance
Nov‐09
Dec‐09
Jan‐10
Feb‐10
05/18/09 06/01/09
05/18/09 05/29/09 05/25/09 06/01/09
Updated Roads Scholar test questions phased in, based on statistical significance and the schedule of upcoming courses.
05/25/09 06/01/09
08/21/09 08/31/09 09/04/09 09/14/09
11/20/09 11/30/09
20
Training Evaluations Level 3 • Survey Monkey • Six months after training • Positive results • Low response rate
21
Review Level 3 surveys and update questions as needed. Convert to paper format. Tell participants at the end of each session that they will be receiving a follow-up evaluation. Send evaluation to participants 3 months after training unless participants would not have had an opportunity to use skills. 22
Update Questions
Task Training ‐ Level 3 Revise Level 3 Evaluation Road Surface Management PennDOT Review Complete First Course Date First Evaluation Implemented Roadway Safety Improvement Program PennDOT Review Complete Evaluation Ready Posting & Bonding of Local Roads PennDOT Review Complete First Course Date First Evaluation Implemented Work Zone Traffic Control Asphalt Roads Common Maintenance Problems PennDOT Review Complete Evaluation Ready Drainage: The Key to Roads that Last PennDOT Review Complete First Course Date First Evaluation Implemented Equipment & Worker Safety PennDOT Review Complete First Course Date First Evaluation Implemented Chainsaw Safety PennDOT Review Complete First Course Date
Please provide details about benefits: Estimated person-hours saved ____ (week) Estimated wage rate for the hours saved ____ (dollars) Estimate other dollars saved ____
28
Benefits of Technical Assistance (Level 4) • Low response rate • Difficult for participants to quantify Recommendation: Conduct interviews or focus groups for the most frequent assistance types (covering approximately 60%).
29
Updated form
3 30
Discussion: Generate examples of workplace actions for as many of the six areas of application as you can. Doing the job correctly Increasing worker productivity Enhancing public safety Enhancing worker safety Saving time in Extending equipment life completing the work
Bonus: Generate examples of workplace actions that do not fit within the six application categories provided.
31
Update Form
PennDOT Review
Task Technical Assistance ‐ One Month Evaluation Revise One Month Evaluation PennDOT Review Complete Evaluation Implemented Technical Assistance ‐ Six Month Evaluation Revise Six Month Evaluation PennDOT Review Complete Evaluation Implemented
