Project
MHeL Millennium Hospital / E-Learning Deliverable / Report
R4
Final report: evaluation of impact and results
Editor
SEGI
Work Package
WP1
Dissemination Level
Public
Status
Final
Date
September 30, 2015
The MHeL Consortium
Beneficiary Number
Beneficiary name
Beneficiary short name
Country
1
Società Italiana di Endoscopia Ginecologica
Italy
2
Imaginary srl
Italy
3
Aristotelio Panepistimio Thessalonikis
Greece
4
Tracoin Quality BV
Netherlands
5
Associazione Ostetrici e Ginecologi Ospedalieri Italiani
6
Societa Italiana di Ginecologia e Ostetricia
Italy Italy
Amendment History Version
Date
Author/Editor
Description/Comments
0.1
01.08.2015
Fabio Gilardenghi
Document Inde
0.2
15.08.2015
Fabio Gilardenghi
First report draft
v1.0
30.08.2015
Fabio Gilardenghi
New sections addition
v1.1
28.09.2015
Fabio Gilardenghi
Finalize draft
V1.2
30.09.2015
Fabio Gilardenghi
Italian translation
Authors Name
Institution
Fabio Gilardenghi
SEGI
Jamil Karim Amirian
SEGI
Reviewers Nome
Istituzione
R.Ackema
TQ
Maria M. Nikolaidou
AUTH
D.Morosini
IMA
Legal Notices The information in this document is subject to change without notice. 2
The Members of the MHeL Consortium make no warranty of any kind with regard to this document, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. The Members of the MHeL Consortium shall not be held liable for errors contained herein or direct, indirect, special, incidental or consequential damages in connection with the furnishing, performance, or use of this material.
Executive Summary This document contains the final evaluation and the report of potential impact on beneficiaries. The purpose of this document is to provide evidence of the evaluation of teaching methods, and the potential for further use in the professional fields of the beneficiaries in order to: -
Confirm the validity of this new teaching method that uses gamification mechanisms, which involves teaching in immersive virtual environments with the help of interactive technology, and which allows for greater engagement by the participants;
-
Measure the impact on the potential mobility of beneficiaries, extending also into the international professional world.
3
Table of Contents 1
Final evaluation report
5
2
Prospective impact on beneficiaries
7
3
Conclusions
26
4
1 Final evaluation report The MHeL project, via the transfer of two best practices of innovation (Mirror and mEducator), has developed, tested and launched a training model of potential impact on the system of continuing education in healthcare. More specifically, the project has modeled a training course for medical specialists, mainly gynecologists, based on the use of an interactive platform that successfully integrates technological
solutions and
methodological standards
into
structured
content
for
competences and learning outcomes, which are accredited for CME recognition. The result obtained from the dual certification and having effectively granted the beneficiaries 600 CME credits in the first trial, and 2598 CME credits in the second, confirmed full adherence of the training model and its contents to the quality standards set forth by the authorities governing the issuance of CME credits. The version of the MHeL platform that underwent testing represents an adaptable and flexible framework for the provision of online educational modules that guarantees a learning experience that is simple, user-friendly and enables the achievement of goals needed for CME credits directly from the workplace. As a result, the healthcare professional will not have to take time away from his/her professional activity, and will benefit from the added advantage of time and cost optimization. The value of the potential impact on the beneficiaries of the project is therefore of notable importance for the future diffusion of this model. The use of gamification and reflection tests is a distinctive feature among the array of training techniques within healthcare because the student is expected to have an active role in his/her personal path of continuing education that is both engaging and motivating. This is possible through the provision of a high-quality, scientific educational experience that is structured with a metadata system that enables its integration into virtual databases built by European university networks. The purpose of this is to increase disclosure and potential impact as part of European certified training. The MHeL project has therefore created a prototype for medical specialization, contributing not only to innovation and the computerization of data in line with the "a new impetus for European cooperation in VET" plan and the flagship initiatives of the EU innovation strategy (Digital Agenda and Innovation in Europe), but also in the creation of an interactive 3D training environment that is both engaging and highly motivating (consistent with the objectives of professional training set forth by the Copenhagen process and the Bruges Communiqué). 5
Finally, by developing an innovative DL model that stands out from amongst the other methods of health care training, this platform has contributed to the facilitation of professional healthcare training according to the European system of accreditation, and therefore promotes employability and mobility. Its creation has also promoted the close collaboration between the most important scientific societies and the Italian gynecological associations, as well as the network of European universities. This has enabled integration between educational, professional and research systems throughout Europe. In terms of operational objectives, the project, in line with what was outlined in the application form: 1. has transferred a best practice technology utilizing game-based and reflective learning within an eHealth context, developed by the European research project Mirro; 2. has developed specialized content that is consistent with the standards defined by the second best practice involved (mEducator); 3. has developed high quality scientific content, created by an internationally renowned opinion leader, and in full harmony with the MHeL methodology and technology, as well as permitted the creation of a training course that ensures CME credits that are valid at both an Italian level and a European one (recognized respectively by the National Commission for Continuing Education in Italy and by the EAACME-UEMS program in Europe), and that responds fully to all quality and essential CME standards set forth by the respective authorities governing the issuance of credits; 4. has tested and standardized a model of learning within an environment that guarantees interactivity, and the provision of relative CME credits.
6
2 Prospective impact on beneficiaries The ongoing evaluations conducted and the final assessments, reported in the R9 and R14 documents, attest to the validity of the training methodology used in the MHeL project, based on gamification processes, which involves teaching in immersive virtual environments with the use of interactive technology to increase participant engagement. These are supported by high quality standards that fully reflect the quality criteria set forth by the Italian National Commission for Continuing Education that regulates specialist training at the European level (EAACME-UEMS). This aspect is clearly evidenced in document R15, where guidelines are drawn that integrate the set of quality standards and the framework elements used with the results of the tests performed. This has enabled the development of recommendations for Italian training providers for using the model within their own portfolios.
2.1 Prospective impact In confirmation of these possibilities and potentials, SEGI, AOGOI and SIGO will be offering the DL course 'Multimodal treatment of cervical cancer' to its associates again beginning January 2016 until December 12, 2016, subject to agreement with MHeL’s technical partners. Given that professional training is one of the strategic incentives of the abovementioned scientific associations, the opportunity was taken to insert - based on the evaluations carried out by the technical MHeL partners in the "micro projection" phase of the training - the DL course "Multimodal treatment of cervical cancer" in the training plan of AOGOI provider ID 2223, enabling free access to all AOGOI, SEGI and SIGO associates, and estimating the participation of at least 1000 beneficiaries (specialists). Below is the screenshot of the insertion of the course into the training plan.
In this eventuality,
gynecologists and other medical specialists in Europe will continue to receive identical training for recognition by the EAACME-UEMS governing authority for up to 2 years from the date of certification, as evidenced by the certificate issued by EAACME_UEMS and published on the home page, http://virtual.mhel-project.eu/client/app/#/login.
7
2.2 Further evaluation of prospective impact by gynecologists, key opinion leaders, methodology beneficiaries and stakeholders – at the same time- as they are potential trainers and scientific managers of future professional training activities 2.2.1 Prospective impact of e-learning on beneficiaries Below is the transcript of the video interviews conducted with industry experts, published in the NEWSROOM section of the project website (http://www.mhel-project.eu/project/news/), which attest to the potential impact of the MHeL methodology on beneficiaries using the online model. The goal of this analysis was to evaluate the teaching methodology and prospective impact on beneficiaries within the Italian university environment and the international environment. Prof. Mauro Busacca
Name Last Name
http://www.mhel-project.eu/project/news/ Qualification
Professor of Obstetrics and Gynecology, University of Milan
Objective of the evaluation
Evaluation of the potential use of the MHeL methodology in a university environment for those specializing in obstetrics and gynecology
8
Contents of the interview
Date of interview
Good morning, my name is Mauro Busacca, Full Professor of Clinical Obstetrics and Gynecology at the University of Milan. This brief intervention serves to highlight the positive experience I had with the Millennium project, from both a teaching and training point of view, as well for the usefulness of what in the future will be online training, not necessarily by a professor or direct. In the Millennium project we had of course the opportunity to access training credits which, as you know, are particularly important for our training. Additionally, these tests, these simulations, these games - as they were defined -, allow us to deal with situations that we often find ourselves faced with in our hospitals, in a way that I would almost describe as a game, as something that allows us to reason and at the same time to see how these are transformed by our individual decisions. I think this is an extremely useful tool, apart from the experience had with Millennium. It can be proposed again in a teaching environment, probably not only for students (perhaps it is a bit too much), but definitely for specialists schools during the last years of training, or in post-specialties master's which instead would need just such tools for knowledge development.
September 28, 2015
Dr. Emilio Arisi
Name Last Name
http://www.mhel-project.eu/project/news/
Qualification
Specialist in Gynecology and Obstetrics. Regional Secretary of AOGOI [Association of Italian Obstetricians and Gynecologists: hospital, territorial and independent professionals] of Trentino Alto Adige, Board Member of the European Society of Contraception and the Board of the European Consortium for Emergency Contraception
Objective of the evaluation
Evaluation of the potential use of the MHeL methodology within national and international training environments, further increasing the catalog of courses for gynecologists.
Contents of the interview
I followed the Millennium training experience to the end, and I must say that I believe it to be very useful and exciting technology for training gynecologists and possibly other specialists from other branches of medicine because, in reality, technology helps and invites you to improve your knowledge of the theme presented. There are sometimes forms of transitional control of learning, which are addressed in a progressive manner over time and in various steps of the training. There is also a final questionnaire that helps to assess the point that has been reached. Technology is actually, let's say, slightly complex at first, but as soon as an attempt is made to enter and "play" with the theme, one actually understands how to move 9
smoothly in this field. The presence of gamification is a plus that allows you to enter more deeply in the learning experience. It involves you and gives you added value compared to other learning experiences. That is why it was particularly appreciated by all those who have dealt with this type of training. I think that this experiment could be successfully used in other branches of medicine, for example in the contraception branch itself, in which I have been venturing for years at both an Italian and a European level.
Date of interview
September 28, 2015
Prof. Chiara Benedetto
Name Last Name
http://www.mhel-project.eu/project/news/
Qualification
Prof. Chiara Benedetto, MD Immediate Past President of the European Board and College of Obstetrics and Gynecology. Chair FIGO Committee on Women's Sexual and Reproductive Rights
Objective of the evaluation
Evaluation of the methodology used and its potential development in gynecology and obstetrics, both in Italy and internationally
Contents of the interview
I just had a look at the new e-learning platform Millennium Hospital. It is a very promising tool, which besides lectures presented by renowned colleagues in the field includes the so called reflection test: aiming to self access the learning effectiveness. Which also includes the gamification, which through the relation of real cases help the users to mimic her/his professional life experience. Because of these features, I feel Millenium Hospital can have a bright future among trainees and specialists in obstetrics and gynecology
Date of interview
September 29, 2015
10
2.2.2 Prospective impact of blended learning on beneficiaries In the interviews that follow, further prospective impacts of the MHeL methodology and technology on beneficiaries are evaluated in advance regarding the "blended" and mixed training - a cutting edge method of training found within the realm of health education. In light of the positive results that already emerged during the first phase of training, SEGI, AOGOI and SIGO have positively assessed the opportunity of possibly using the MHeL methodology in a blended format for gynecologist training, to be carried out in the future and at the appropriate locations, during events that are already planned for the residential "macro-planning" program for 2016, and in agreement with all the technical MHeL partners. More specifically, courses that are already scheduled must be evaluated in relation to training activities by the AOGOI provider, which could benefit from the use of MHeL technology and methodology to facilitate more effective teaching and a greater involvement of the participants in residential programs. Two different methods have been proposed: o deferred in reference to the residential program, and therefore in a "blended" mode for the asynchronous transfer of practical exercises to the MHeL platform, where it could be possible to access the CME learning tests from the entire program, as well compile the satisfaction questionnaire, both remotely and online, with significant reduction in necessary resources for activities to be carried out during the program. This could also provide the additional benefit to beneficiaries of being able to test their knowledge using simulation or gamification processes offered by the MHeL methodology in an appropriate location, and with themes suited to individual needs, without having to leave the workplace any more than is necessary. This guarantees the user helpful interaction with his/her instructor/tutor, as proposed by the MHeL model. o In "mixed" mode with the use of ad hoc telematic instruments for the room (e.g. with the aid of remote voting). With the possible use of remote voting directly in the classroom, participants may follow the video lessons on the MHeL platform in a synchronous manner, listening to the lecturer and completing the exercise via a televoting system, thus ensuring their active participation and a high level of interaction with each other and with the instructors. Learners will be able to interact with the speaker, for example, giving real-time answers about the most suitable treatment plan (pharmacological or surgical), therapy to be administered, etc. Following are the transcripts of the interviews given by the scientific managers of the events. These events, already scheduled in the training plan of the AOGOI provider, were deemed potentially suitable for the future use of MHeL methodology according to what was stated previously, and in agreement with all the technical MHeL partners. It is understood that in the "micro-planning" phase of the individual CME course, the designated scientific manager of each program will define the architecture of the training, including scientific content and the educational model of the project. In this planning phase, the scientific manager will need to evaluate any changes to the medical-scientific reference context, working together with the MHeL technical manager (for the inevitable assessments of the multimedia platform and any opportunities for improvement in the technological sector) and the MHeL organizational manager (for the inevitable assessments of economic sustainability and organization of the course with respect to the available budget and the educational expectations of possible participants). 11
It is the responsibility of the scientific manager to inform all partners of the progress of the course planning through stages of design review and verification, promoting planning choices that are made to ensure that the results of the planning process are consistent with the needs of the learners. Note: The questions posed during the following interviews with individual respondents, who represent opinion leaders in gynecology and scientific managers of the events mentioned above, are given below:
1) How would you rate the use of the MHeL methodology in gynecologist training?
2) Do you think that this training tool could be used with other, more traditional training methods?
3) What are the advantages? Dr. Claudio Crescini
Name Last Name
Qualification and short presentation
Specialist in Obstetrics and Gynecology, Department Director Obs Gyn Hospital of Treviglio (BG) Maternity and Children's Department, National Secretary of SEGI, AOGOI Regional Manager of Lombardia, Member of the AOGOI Scientific Committee ID 2223
Objective of the evaluation
Evaluation of the potential added value on the effectiveness of training and the prospective impact on beneficiaries using a teaching methodology that integrates two diverse teaching methods (blended RES + DL). As a member of SEGI for several years and National Secretary since May 2015, I was able to actively participate in the various stages of the creation of the "Multimodal treatment of the cervical cancer" course, also providing support to the online attendance of the doctors from my hospital who eagerly participated in the course, accredited by the AOGOI provider from August 8 to September 29, 2015. I was thus able to personally experience the value of an education that is based on the concept of 'learn by doing,' and appreciate the effectiveness of such training.
Contents of the interview
I therefore believe it very advantageous - and I say this as the AOGOI Head of the Lombardia region and as a member of the Scientific Board of the AOGOI provider - to use this method on a broad scale for the delivery of distance learning courses that touch on all issues of interest for gynecologists, especially those working in a hospital. The setting of the Millennium hospital provides an effective 'model of the professional experience' and learning localized within the work environment. This also provides the added advantage of not having to go somewhere else in order to receive the same CME credits, which are also recognized by the European Authority (UEMS), as well as a lower cost. In light of these assessments you could consider including the MHeL methodology and technology to support the residential training of upcoming programs on "Obstetrician Emergencies in the Delivery Room" scheduled for 2016, and subject to agreement with the MHeL project partners.
12
Using this method, we might effectively empower autonomous learning (albeit supported by the constant presence of a tutor to be contacted via e-mail) for the entire practice by inserting films that are relevant to the delivery room regarding primary actions to be taken in case of an emergency, and associating reflection tests to these films in order to reinforce the topics learned and the themes covered, as well as gamification processes for greater participant engagement. The use of the Millennium platform also offers the added benefit of allowing the completion of the tests covering the topics addressed and the completion of a satisfaction test directly online, greatly reducing time and cost spent in the classroom. This would enable us to greatly innovate professional teaching in Italy, and to get increasingly closer to a training model based more on outputs, as the European one is.
Date of interview
September 14, 2015 Employment opportunities -to be evaluated with all the MHeL technical partners- are evaluated as part of the course, which is already included in the AOGOI 2016 Training Plan. See screenshots below.
Outcomes of the evaluation of the potential impact
Dr. Giuseppe Ettore
Name Last Name
Qualification and short presentation
Department Director Gynecology and Obstetrics ARNAS Garibaldi Nesima - Catania, AOGOI Regional Manager Sicily
Objective of the evaluation
Evaluation of the potential added value on the effectiveness of training and the prospective impact on beneficiaries using a teaching methodology that integrates two diverse teaching methods (blended RES + DL).
Content of the interview
Having supported firsthand the training of many doctors in my region who took and completed the course "Multimodal treatment of cervical cancer," I was able to appreciate how easy it was
13
to benefit from the lectures, and the quality of the surgical videos, which very educational and of high scientific value. But what struck me the most were the interactive sections of the "reflection test" and the gamification process," that aside from facilitating a "participatory attendance," also enabled more effective learning and better retention, as well as creating a learning atmosphere that was very similar to the clinical and surgical practices that we gynecologists see every day. I think that the Millennium Hospital platform is a cutting-edge interactive learning environment that, despite needing some small improvements, is well suited to be used in combination with our training, especially in fields such as gynecological surgery, that will be the topic of the next residential programs taking place in Sicily. And it is for this reason that efforts should be made to integrate the MHeL model with the usual model of interactive residential training. As an aid to the execution of courses, all the videos of the latest gynecological surgery techniques could be uploaded to the MHeL platform, supported by games that simulate the surgical experience. This could facilitate a more immersive experience for gynecologists that is based primarily on outputs. Being able to delegate all verifications and approvals to the MHeL platform would constitute an additional added value in terms of time spent in the program venue, with consequent cost savings. By working in this manner, we could also increase the overall impact on the method of professional training, fulfilling the desired requirements recently issued by the National Commission for Continuing Education in regards to using blended methods. Date of interview
September 14, 2015 Employment opportunities -to be evaluated with all the MHeL technical partners- are evaluated as part of the course, which is already included in the AOGOI 2016 Training Plan. See screenshots below.
Outcomes of the evaluation of the potential impact
Dr. Franco Marincolo
Name Last Name
14
Qualificatio n and short presentatio n
Department Director Human Reproductive Pathophysiology and PMA Center Level I Hospital G. Campagna - via Rimembranze,1 87064 CORIGLIANO CALABRO (CS), AOGOI Regional Director Calabria
Objective of the evaluation
Evaluation of the potential added value on the effectiveness of training and the prospective impact on beneficiaries using a proven methodology with Millennium Hospital aided by technological tools in the room (e.g. remote voting). After having participated directly in the Millennium course I realized the obvious advantage that these new teaching methods can bring to medical education, even in a mixed form. The MHeL methodology that was recently tested (with a high level of participation in Calabria), would equip us to accept the challenge of creating innovative training that can be integrated into a training program, with the obvious advantage of simulation systems that can be of great help in understanding specific cases, for example, within the medical-legal context, that today are increasingly common in everyday clinical practice.
Contents of the interview
The goal would therefore be to treat the more practical aspects of technical consultancy in proceedings related to civil and criminal liability for damages and medical care within an interactive platform that could be set up within Millennium Hospital. The integration of a method that enables the immediate assessment of learning and reinforces the efficacy of training using simulations, greater participant involvement and "fun" techniques, could be of paramount importance, especially in this field. This represents a real added value in the quality of professional training in medical-legal material, and enables us, the trainers, to use an innovative method while still remaining within a residential setting where instructors and tutors can continuously interact with the participants. The verification of learning, the involvement of the content and MHeL interaction - which can be projected in the classroom - could be done immediately with the help of traditional televoting devices and ongoing interaction between instructors/tutors and participants.
Date of interview
September 14, 2015 Employment opportunities -to be evaluated with all the MHeL technical partners- are evaluated as part of the course, which is already included in the AOGOI 2016 Training Plan. See screenshots below.
Outcomes of the evaluation of the potential impact
15
3 Questionnaires given to gynecologists during residential training during the period of September 15-30, 2015 (external evaluation) 3.1 Objective and Strategy In order to further confirm the validity of this new teaching method (that utilizes gamification processes, which promote learning in immersive virtual environments with the help of interactive technology, allowing for greater engagement by the participants), it was decided to carry out an external, parallel evaluation with gynecologists who did not participate in the experimentation, in order to further gauge the potential impact on beneficiaries, especially in view of increased mobility within the international workplace.
3.2 Methodology The above-mentioned assessment involved a large group of gynecologists. This was made possible given the large presence of gynecologists as a result of residential programs carried out in Milan between September 15-30, 2015. 3.3 The online survey targeted to gynecologists In order to carry out this assessment, an online survey area was implemented into the MHeL platform. The details are given below. The number of gynecologists reached was 125. The online invitation http://virtual.mhel-project.eu/client/app/#/share
The presentation of the MHeL video tutorial
16
Access to the questionnaire
Personal information requested
All items required by the questionnaire Name:
Surname:
E-mail:
17
Gender:
Female
Male
Birthday:
Job province:
Profession:
Medical sector:
Association membership:
Main working environment:
Hospital
University
Private practice
Other
Was it easy to go through the content of the course?
Absolutely no
18
No
Neither yes nor no
Yes
Absolutely yes
Was the quality of the entire virtual training solution acceptable?
Absolutely no
No
Neither yes nor no
Yes
Absolutely yes
Does the gamification mechanism assure benefits, comparing to the traditional training solutions?
Absolutely no
No
Neither yes nor no
Yes
Absolutely yes
Do you think your competencies will be improved thanks to this virtual training solution?
Absolutely no
No
Neither yes nor no
Yes
Absolutely yes
Do you think this training tool could be transferable to your medical sector?
Absolutely no
No
Neither yes nor no
Yes
Absolutely yes
Would you like to have this training tool for your medical training?
Absolutely no
19
No
Neither yes nor no
Yes
Absolutely yes
Would you suggest this training tool to your colleagues?
Absolutely no
No
Neither yes nor no
Yes
Absolutely yes
How would you improve this virtual training solution?
I declare I have read the MHeL project methodology and its navigation mode.
confirm
1.3.6 Questionnaire : Submission
20
3.4 Concluding remarks on the evaluation and assessment of impact on beneficiaries, outlined as a result of the investigation carried out in parallel with the experimentation Positive results were also collected from the online survey addressed to beneficiaries, which was held in parallel with respect to 2nd pilot transfer, as showed below. From the analysis of question n. 15 it emerged that the introduction to the course / navigation guide is an useful guide for understanding the navigation model (92 % gave a favorable opinion). Q15: Did the introduction give you sufficient information to understand the use of the navigation? Q15
Absolutely yes Yes Neither yes nor no No Absolutely no
50 37 7 0 0
53,19% 39,36% 7,45% 0,00% 0,00%
(Abs) Yes 92,55%
94
97 % answered that the training objectives were clearly defined
Q16: Are the learning goals clearly defined?
Absolutely yes Yes Neither yes nor no No Absolutely no
58 33 3 0 0 94
61,70% 35,11% 3,19% 0,00% 0,00%
(Abs) Yes 96,81%
21
The navigation method to use of content was evaluated positively by more than 90 % of participants. As a matter of fact, to question. n. 17 emerged that it's easy to go through the contents of the elearning module.
Q17: Was it easy to go through the content of the course? Q17 Absolutely yes Yes Neither yes nor no No Absolutely no
59 26 9 0 0 94
62,77% 27,66% 9,57% 0,00% 0,00%
(Abs) Yes 90,43%
The Reflection tests were experienced as a real benefit from more than 90% of the users, as shown by the results of question 18. 22
Q18: Were the self-evaluations helpful to support the knowledge acquisition? Q18 (Abs) Yes Absolutely yes 49 52,13% 90,43% Yes 36 38,30% Neither yes nor no 9 9,57% No 0 0,00% Absolutely no 0 0,00% 94
The results from question n. 21 showed that the notepad has proved to be an effective tool during the learning experience (93.62%).
Q21: Was the notepad useful during the course? Q21 Absolutely yes Yes Neither yes nor no No Absolutely no
62 26 6 0 0 94
65,96% 27,66% 6,38% 0,00% 0,00%
(Abs) Yes 93,62%
23
According to the 85% of the participants, the educational benefit that is obtained through the use of gamification is significant. Q23: Does the gamification mechanism assure benefits, comparing to the traditional training solutions? Q23 (Abs) Yes Absolutely yes
51
54,26%
Yes
29
30,85%
Neither yes nor no
14
14,89%
No
0
0,00%
Absolutely no
0
0,00%
85,11%
94
24
Over 90 % believe that this training method is transferable to other medical disciplines (v . results with respect to question n. 25).
Q25: Is this training tool transferable to other medical sectors? Q25 (Abs) Yes Absolutely yes 59 62,77% 91,49% Yes 27 28,72% Neither yes nor no 8 8,51% No 0 0,00% Absolutely no 0 0,00% 94
25
3 Conclusions Evaluation of the platform carried out in the project The analysis of the 1st and 2nd Trasfer pilots fully described in R9 and R14, confirm the assumption that the blending of “game based learning” combined with “reflective e-learning” results in a sustainable solution that can be effectively used in all fields of medical sector. This was also confirmed by receiving both accreditations per ECM and CME credits. Finally we conclude that the MHeL solutions can be utilized as an adaptable one-stop learning solution for any sector where large groups of learners can benefit from the knowledge and experience of experts in their field of working. Potential impact on the professional training system (players and decision-makers) The consortium’s willingness aims to consider the results achieved through the MHeL project as a common asset, in light of synergistic relationships aimed at improving the courses offered in the healthcare sector, including those provided using innovative methodologies. As a result, a list of recommendations to be addressed to the main players were prepared based on (a) the analyses performed within the project and summarized in other results (R9 and R14) and (b) the evaluation by training operators who were directly involved. In this regard, the learning results acquired directly by the consortium operators (physicians) who created the training model and programs is a relevant experience in the training system, since there is a chance to explore its limits and potential. The Millennium Hospital e-Learning project has shown to have characteristics for which it can become a tool for CME Providers to deliver training courses in Italy in compliance with the regulatory requirements described in the State-Regions Agreement dated April 19, 2012. The platform was, in fact, implemented as required by law in Italy and Europe, allowing to manage (a) user registration and identification, (b) training program tracking, (c) completion of the assessment questionnaires on the program learning results and evaluation, (d) download of the certificate showing the CME credits earned, and (e) retrieval of data that must be sent to the Authorities responsible for the final accreditation of CME credits. It also allows to fulfill the obligation imposed by the CME regulations to "trace" the learner's path and, therefore, allows the Provider to assign CME credits exclusively to participants who actually meet the requirements, in compliance with the ministerial requirements as well. The interactive nature of the training tool is to be considered an additional added value to the future development of continuing medical education, as well as an additional quality aspect of the model that can be transferred to future CME programs, for the distance learning (DL) programs, as well as for the blended activities.
Potential impact on beneficiary gynecologists and other specialists 26
The results based on the 1st and 2nd Transfer Pilots, the external online survey, as well as on interviews to key opinion leaders and to the major ECM players, positively underline the potential transferability of the methodological and technological innovations within the various specialties that make up the “health population”, and to evaluate its integration with other typical learning systems in the field of medical science. We have seen as MHel project made it possible to combine two axioms once hardly joinable: the need for economic and organizational sustainability (i.e. train a large number of health professionals through the ECM course), as well as the andragogic need (i.e. how learners consider training an active and engaging knowledge seeking process, not merely contents reading). The contributions collected showed not only the potential impact on beneficiaries (gynecologists) considering the possibility to organize several courses based on the MHeL method, as well as the use of the MHeL technology supporting residential activities to be held in the field of obstetrics and gynecology, but that the MHeL method is also applicable to other medical disciplines and therefore is not only conceivable for the field of gynecological microsurgery. The methodology, in fact, is perfectly adaptable to all stages that make up a health treatment (anamnesis management, delivery of health care, evaluation of results, etc.) because it can propose the management of a clinical case (real or simulated) to participants and correlate it with clinical documentation, laboratory examinations, consultation and reporting by other medical specialists. As we have seen, nowadays medical education reflects traditional methods based on the descendant and non-interactive model, where student participation is limited to moments of discussion during the teacher-led classes. In general, the proposed training is passive – mainly based on inputs- and this induces the doctor to assume a predominantly receptive position on the proposed contents. But MHeL radically changes this situation by proposing a flexible summary that integrates the most important aspects of the two modes (traditional training based on inputs with interactive training based on outputs). In an immersive environment and through fun activities, doctors were able to feel part of a process that they themselves realized requires initiative, self-management, and exploratory activation. The contents and skills proposed are recognized and valid, as well as encoded in a manner consistent with European alignment systems, and represent the frontiers of research translated in terms of training by high-level experts. The environment is rather identical to the personal experience of self-training, in which the protagonist is above all the interaction with the other doctors, even if virtual. Doctors have therefore rediscovered their usual training experience, which is however validated and in dialogue with the European professional and healthcare community. In this sense, MHeL radically reduces the increasingly important gap between an research that advances at progressively higher speed and the multiplicity and complexity of the work of the single physician, compared with often contextual problems and with a competent consumer who has access to many information sources, once unavailable.
27
The model proposes and candidates as a tool for enhancing comparison, constant learning, system-making of knowledge among physicians in Europe, interest of transparency and official coding, to protect patients. The transferability of the model is derived from the foregoing references, characteristics described are common to all specialties, and more broadly relationship between the medical profession, training models, and validation.
exchange, but in the since the affect the
Potential impact on the European training system In recent time, the established methods of continuing education for the health professions have been largely ineffective in helping medical professionals maintain their clinical knowledge and competence. To be most effective, distance medical education should be a seamless part of the clinician’s work environment. Instead of requiring a medical learner to leave a clinical encounter to seek learning, opportunities will be embedded within clinical systems, delivering the right information and education at the time of need. For learners seeking more comprehensive information after a clinical encounter, medical education must be ubiquitous, available at any time from any Internet-connected device to which they have access. Clinical learners of all types/levels (undergraduate, post-graduate, continuing) should receive credits for both types of activities without having the administrative burden of printing a certificate and submitting it to the appropriate certifying body. With the ability to track educational activity, educators will be able to conduct outcomes studies to determine the effectiveness of medical education and its impact on clinician behavior and patient care. To that respect, the MHeL project has done a good step forward in that it has tried to combine – with evidence of positive results- different approaches and technological standards that are already enabling the seamless integration of the technological platform (i.e. the virtual media-rich environment) with the curriculum design process and ready-toteach practice, based on the re-usage of contents, which is one of the milestone of mEducator. An ultimate goal of the mEducator metadata description scheme was to provide the aforementioned standardized format in order to enable the medical educational resources to be shared, exchanged, searched and retrieved across academic institutions. This is actually the envisaged hope for MHeL's sustainability too: the provision of metadata schemas and standards for describing modules in an innovative environment would allow repurposing actions and would essentially improvise for the continuation and inclusion of the MHeL course in different programs as well as different contexts (both thematically and technically speaking). The latter can also be applied in train-the-trainer course as well. To this extent, the long term aim is to reach a modernization of medical curricula through also training the Universities staff in innovative teaching methods and specific pedagogical approaches. With such a strategy, Medical Schools can play pivotal roles in improving the quality of healthcare; in the long run, the MHeL project will benefit not only the involved target groups, but also the wider society at local, national and global levels. 28
So it is believed that the undertaken project activities will contribute to the establishment of a network of institutions contributing to the enhancement of MD education. Its close links with scenario based approaches and problem based learning (PBL, VPs) enhance its capacity to modernise professional curricula.
Potential impact of possible developments in terms of model innovation MHeL project aims to gain potential improvements both in terms of training platform review and training design. As a matter of fact, in addition to the constant technological upgrading, required by the natural evolution of the technological equipment used by the learners, there is another aspect that should be considered, that is the possibility of structuring the MHeL platform even when the outcomes and results achieved are managed, with the possibility of recontacting the professionals who participated in the course. The purpose is to make professionals go back to the platform and bring their testimony on how their job changed thanks to what they learned and experienced in the MHeL course. This option would also encourage a constant cultural and professional exchange between participants and maintaining an active community. In a working group that hopefully will be in connection with the institutions, the integration of the MHeL platform with the computer systems of the CME data inspecting authority in Italy (AGENAS) could be also be examined.
29
