MHeL Millennium Hospital / E-Learning Deliverable / Report
R14
Report on the 2nd Pilot transfer – evaluation report Editor
TQ
Work Package
WP6
Dissemination Level Public Status
Final
Date
30-09-2015
MHeL
Report on the 2nd Pilot transfer – evaluation report
Page 2
The MHeL Consortium Beneficiary Number
Beneficiary name
Beneficiary short name
Country
1
Società Italiana di Endoscopia Ginecologica
Italy
2
Imaginary srl
Italy
3
Aristotelio Panepistimio Thessalonikis
Greece
4
Tracoin Quality BV
Netherlands
5
Associazione Ostetrici e Ginecologi Ospedalieri Italiani
6
Societa Italiana di Ginecologia e Ostetricia
Italy Italy
Version 1
MHeL
Report on the 2nd Pilot transfer – evaluation report
Page 3
Amendment History Version
Date
Author/Editor
Description/Comments
0.1
01-09-15
R. Ackema
Based on analyses V4
0.2 / 03
27-09-15
R. Ackema
Updated based on review results
1
30-09-15
R. Ackema
Final version.
Contributors Name
Institution
R. Ackema
TQ
Maria Nikolaidou
AUTH
Reviewer Name
Institution
J. K. Amirian
SEGI
D. Morosini
IMA
Legal Notices The information in this document is subject to change without notice. The Members of the MHeL Consortium make no warranty of any kind with regard to this document, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. The Members of the MHeL Consortium shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material.
Version 1
MHeL
Report on the 2nd Pilot transfer – evaluation report
Page 4
Executive Summary As progress in medicine becomes faster, the necessity to up-date professional knowledge is increasing, as well as demands for a better anticipation of skill needs and labour-market. The sectorial fragmentation and the week comparability and recognition of qualifications gained by doctors throughout Europe have a negative impact on their mobility, competitiveness and potential employability. Different qualifications systems discourage continuous training in other countries and limit the mobility of doctors. Due to the above sectorial fragmentation, low level of investments, international co-operations and exchanges of good practices limits the development of the sector. Despite eHealth has been proved effective in reducing market fragmentation through benchmarking, standardization and certification, low investments and weak public-private cooperation still inhibit the integration of technological innovation. According to this framework, The Millennium Hospital e-Learning (MHeL) project fosters synergies between professional associations, training providers, certification bodies and ICT developers. In Italy, the aim is to adapt the CME in minimal invasive surgery, innovations, quality standards and solutions developed in 2 eHealth European best practices. More in details, this document describes: The evaluation approach and method the results of the 2nd Pilot transfer with 543 users (521 Italians and 22 Europeans) participated in the evaluation of the MHeL platform and the e-learning program: Multimodal treatment of cervical carcinoma;
Conclusions and lessons learned.
The 2nd Pilot transfer showed a very high satisfaction rates from the users:
83% of participants reported that the topics discussed in the course were (very) relevant in their daily practice;
Almost all the users (96%) improved their competences thanks to this virtual training solution.
96% of users believe that this training tool is transferable to other medical sectors.
In an additional evaluation carried out with “external users”, 93% of the users did report that they believe that this training tool is transferable to their medical sectors The results of the 2nd Pilot transfer confirm the assumption of the MHeL project that the blending of “game based learning” combined with “reflective e-learning” results in a sustainable solution that can be effectively used in the medical sector. This was also confirmed by receiving both accreditations. We conclude that the MHeL solutions can be utilised as an adaptable one-stop learning solution for any sector where large groups of learners can benefit from the knowledge and experience of experts in their field of working.
Version 1
MHeL
Report on the 2nd Pilot transfer – evaluation report
Page 5
Table of Contents 1
Introduction
8
2
Evaluation approach and model
9
3
4
5
6
7
2.1
Evaluation phases
9
2.2
Evaluation model
10
Evaluation results of the 2nd Pilot transfer
11
3.1
General
11
3.2
Results of the 2nd Pilot transfer
11
3.3
Results of the evaluation of “External Others
12
Detailed information of the evaluation results
13
4.1
Demographic data
13
4.2
Level 1 – Reaction / usage
17
4.3
Level 2: Learning
27
4.4
Level 3: Behaviour – subjective improvement in work
32
4.5
Analyses of the final test results
37
Data Analysis for “External Others”
40
5.1
Overview
40
5.2
Demographic data
40
5.3
Evaluation of the Quality
41
5.4
Transfer
44
Conclusion and lessons learned
48
6.1
Conclusion
48
6.2
Lessons Learned
49
Appendix 1: Evaluation Toolkit
50
Version 1
MHeL
Report on the 2nd Pilot transfer – evaluation report
Page 6
Table of Figures Figure 1: Birth nation ............................................................................................................13 Figure 2: Overview cities ......................................................................................................14 Figure 3: Profession .............................................................................................................17 Figure 4: Employment status ................................................................................................17 Figure 5: Course viewed.......................................................................................................18 Figure 6: Mini games ............................................................................................................18 Figure 7: Score intro test ......................................................................................................19 Figure 8: Main work environment .........................................................................................19 Figure 9: Introduction on navigation......................................................................................21 Figure 10: Easy to go through content ..................................................................................22 Figure 11: Quality of the videos ............................................................................................23 Figure 12: quality of the entire virtual training .......................................................................24 Figure 13: notepad usefulness .............................................................................................25 Figure 14: average loading time of the videos ......................................................................26 Figure 15: educational quality of the CME program ..............................................................27 Figure 16: Quality of teaching ...............................................................................................28 Figure 17: learning goals clearly defined ..............................................................................29 Figure 18: self-evaluations helpful ........................................................................................30 Figure 19: gamification mechanism assure benefits .............................................................31 Figure 20: relevance of the topics to daily practice ...............................................................32 Figure 21: Usefulness of this E-leaning ................................................................................33 Figure 22: Improved competencies.......................................................................................34 Figure 23: tool transferable to other medical sectors ............................................................35 Figure 24: Attempts-evaluation .............................................................................................37 Figure 25: Test attempt - profession .....................................................................................38 Figure 26: Intro Test- Score Evaluation Test ........................................................................38 Figure 27: Earned Prizes- Score Evaluation Test .................................................................39 Figure 28: profession ............................................................................................................40 Figure 29: Medical sector .....................................................................................................41 Figure 30: Main working environment ...................................................................................41 Figure 31: Ease to go trough the course ...............................................................................42 Figure 32: Quality of the entire virtual training solution .........................................................43 Figure 33: Benefits ...............................................................................................................44
Version 1
MHeL
Report on the 2nd Pilot transfer – evaluation report
Page 7
Figure 34: Transfer ...............................................................................................................45 Figure 35: Training tool for medical training ..........................................................................46 Figure 36: Suggest to collegues ...........................................................................................46
Version 1
MHeL
Report on the 2nd Pilot transfer – evaluation report
Page 8
1 Introduction This report describes the results of the 2nd transfer of the project where (mainly) Italian doctors / physicians – mainly gynecologists as requested by the project- have conducted the training as provided within the MHeL platform. Based on their course results doctors / physicians can apply for certification of skills promoted contemporary at Italian and European level. The objective of the 2nd transfer was to determine, for a large number of users, the impact of the learning experience provided for minimally invasive gynaecological surgery through the MHeL platform and e-learning program: Multimodal treatment of cervical carcinoma. During the 1st Pilot transfer we analysed the results / feedback of 100 users. During this 2nd Pilot transfer we analysed the results / feedback of 543 users, the ones who completed the MHeL course. However more than 1000 users registered themselves on the platform. The evaluation is based on the Kirkpatrick’s model for evaluations of learning experiences and is in detail described in deliverable R9 (Report on 1st Pilot transfer). This includes a dedicated toolbox for supporting the evaluation. The summary of the results of the evaluation of the 2nd Pilot transfer are described in chapter 3. The detailed information of the evaluation is presented in chapter 4. The 2nd Pilot transfer confirmed the very high satisfaction rates from the users as observed during the 1st Pilot Transfer:
78% of the participants reported that the topics discussed in the course were (very) relevant in their daily practice;
92% of the users improved their competences thanks to this virtual training solution; 93% of users believe that this training tool is transferable to other medical sectors.
In an additional evaluation carried out with “external users” (non gynecologists), 93% of the users did report that they believe that this training tool is transferable to their medical sectors. (See also chapter 5) The conclusions and lessons learned are described more in detail in chapter 6.
Version 1
MHeL
Report on the 2nd Pilot transfer – evaluation report
Page 9
2 Evaluation approach and model The main goals of the evaluation were: 1. to improve the MHeL platform and e-learning program during the System Testing and 1st Pilot transfer; 2. to gain a clear and solid evidence of the impact of the MHeL platform and e-learning program during the 2nd Pilot transfer.
2.1 Evaluation phases 2.1.1
System Testing
During system testing the MHeL platform and e-learning program were tested, by representatives of the project partners, to verify if the system meets the specified requirements. (This has been reported in R9 MHeL 1st Pilot transfer) 2.1.2
1st Pilot transfer
SEGI had selected a group of medical specialists to evaluate the MHeL platform and e-learning program. Experts from the project partners provided guidance during this evaluation. Where applicable improvements were made to the MHeL platform and e-learning program. The results of the evaluations of the 1st Pilot transfer did prove objectively that the MHeL system and e-learning program were mature and robust enough to start the 2nd Pilot transfer (the results have been reported in R9 MHeL 1st Pilot transfer). 2.1.3
2nd Pilot transfer
After both the MHeL platform and the e-learning program had reached good maturity it was time to move to large scale of use in an extremely short period of time. The goal was no longer to optimize, but to measure the impact of the learning experience. For this a larger number of users were needed to reach statistical reliability. SEGI asked volunteers / experts to complete the full course on “Multimodal treatment of cervical carcinoma”. At the end of the course we asked the users / learners to fill in a questionnaire about the usability and perceived added value of MHeL method. The course participants were also be asked to evaluate the quality of the e-learning program (as indicated in “Regolamento applicativo dei criteri oggettivi di cui all’Accordo Stato-Regioni del 5 Novembre 2009 e per l’accreditamento” approved by the CNFC on 13 January 2010). The questions used during the 2nd Pilot transfer are in detail described in Appendix 1 of this document (see chapter 7). MHeL is accredited both within Italy and at the European level. The evaluation by the aforementioned physicians, was held according to the Italian regulations for accreditation, as well as to the essential criteria designed by the European authority EAACME-UEMS. Participants were tested for their knowledge on a multiple choice exam (as described in “Criteri per l’assegnazione dei crediti alle attività ECM” of 13 January 2010, page 15, note 2, and “Regolamento applicativo dei criteri oggettivi di cui all’Accordo Stato-Regioni del 5 Novembre 2009 e per l’accreditamento approvato dalla Commissione Nazionale per la Formazione Continua” of 13 January 2010, page 25, note 7).
Version 1
MHeL
Report on the 2nd Pilot transfer – evaluation report
Page 10
All those who passed the examination with at least 75% correct responses received a certificate of attendance that indicates the number of credits acquired as regulated by the Accordo Stato-Regioni of 19 April 2012 and its attachment 1 “Linee Guida per i manuali di accreditamento dei provider nazionali e regionali/province autonome” and the European accreditation.
2.2 Evaluation model In line with the levels of evaluation developed by Kirkpatrick, we will consider the first three levels of summative evaluation. 1 Because MHeL platform supports serious game based learning (rather than Kirkpatrick’s formal learning) we have modified these levels slightly, as described below Level 1: Reaction To what degree do participants react favourably to MHeL approach of learning? Reactions to instances of the learning are measured. Because MHeL is focussing on games based, informal learning, Level 1 additionally includes whether participants are motivated to use the MHeL platform. This can be measured by recording when and how this is used (i.e., with log files). In addition, we can evaluate the user experience through subjective evaluations. There is much overlap with formative evaluation regarding this level, but in the summative evaluation we focus more on usage as a precondition for the next levels. Level 2: Learning To what degree do participants acquire knowledge, skills, attitudes, confidence, and commitment? Level 2 is concerned with what users learn, and how that learning is aligned with what we intend for them to learn. Learning can be assessed by specific evaluation forms that ask for documentation of learning outcomes or subjective ratings. Quantity and quality of these learning outcomes can then be examined. Level 3: Behaviour To what degree do participants apply what they learn? Assuming learning has occurred, and that the work environment is conducive, it is now possible to measure the conversion rate of behavioural intentions into actions. Level 3 is concerned with whether the new knowledge and skills are put to use.
1
(Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating trainings programs: The four levels. 3rd Edition. Berrett-Koehler Publishers, Inc.)
Version 1
MHeL
Report on the 2nd Pilot transfer – evaluation report
Page 11
3 Evaluation results of the 2nd Pilot transfer 3.1 General Based on the results of the 1st Pilot transfer (see R9) it was decided that the MHeL platform and the e-learning program had reached good maturity and the project could move immediately to the 2nd Pilot transfer for a large scale of use evaluation to gain a clear and solid evidence of the impact of the MHeL platform and e-learning program. Individuals who had applied for acquisition of skills of microsurgery and volunteers / experts invited by SEGI conducted the full course on “Multimodal treatment of cervical carcinoma”. The course participants have been also asked to evaluate the quality, usability and potential impact of the e-program as indicated in “Regolamento applicativo dei criteri oggettivi di cui all’Accordo Stato-Regioni del 5 Novembre 2009 e per l’accreditamento” approved by the CNFC on 13 January 2010. The evaluation by the aforementioned physicians, was held according to the Italian regulations for accreditation. Participants are be tested for their knowledge on a multiple choice exam with double randomisation (as described in “Criteri per l’assegnazione dei crediti alle attività ECM” of 13 January 2010, page 15, note 2, and “Regolamento applicativo dei criteri oggettivi di cui all’Accordo Stato-Regioni del 5 Novembre 2009 e per l’accreditamento approvato dalla Commissione Nazionale per la Formazione Continua” of 13 January 2010, page 25, note 7). All users who passed the examination (533 users) with at least 75% correct responses received a certificate of attendance that indicates the number of credits acquired (as regulated by the Accordo Stato-Regioni of 19 April 2012 and its attachment 1 “Linee Guida per i manuali di accreditamento dei provider nazionali e regionali/province autonome”).
3.2 Results of the 2nd Pilot transfer 3.2.1
Demographic data
In total 543 users participated in the 2nd Pilot transfer. More detailed demographic data is described in chapter 4.1. 3.2.2
Level 1: Reaction (Usage)
We asked the users questions about the usability of the MHeL platform and the content of the lessons. Log files have been analyzed to provide the usage data. Summary of the results: 1. All users of “Group A” viewed all the modules/lessons and successfully completed the mini games. 2. The majority of the users (82%) believe that the introduction gave them sufficient information to understand the use of the navigation. 3. 92% of the users accepted the quality of the entire virtual training solution. 4. The quality of the videos was acceptable according to 82% of the users. 5. The average loading of the videos was fast (
