INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Designing Level 3 Evaluations for Continuing Education Programs Self Report and Observation of Practice Mary McAdams, RN-BC, MEd Professional Development Specialist, The Norman Knight Nursing Center for Clinical & Professional Development Massachusetts General Hospital [email protected]

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

DISCLOSURE TO LEARNERS: I have no conflicts of interests to disclose

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Program Objectives • Compare and contrast 2 different methods of level 3 evaluation • Describe 5 levels of evaluation

• Describe 2 reasons why the Professional Development Specialist should gain expertise in the design of level 3 evaluations

INSPIRATION | INNOVATION | TRANSFORMATION

BACKGROUND THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

NPD Standard 3: Outcomes Identification Criteria include: •Involve stakeholders in choosing outcomes for CNE •Uses outcomes to demonstrate programs are valuable and have impact

INSPIRATION | INNOVATION | TRANSFORMATION

BACKGROUND THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Magnet Exemplary Practice 15 Performance reviews that include: •Self appraisal (self report) •Peer feedback (observation of practice) •Clinical nurses and nurse leaders

INSPIRATION | INNOVATION | TRANSFORMATION

BACKGROUND THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

American Nurses Association Code of Ethics, Provision 5.2

“Evaluation of one’s own performance, coupled with peer review, is a means by which nursing practice can be held to the highest standards...“

INSPIRATION | INNOVATION | TRANSFORMATION

BACKGROUND THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Models: Philips ROI

Kirkpatrick

Moore

INSPIRATION | INNOVATION | TRANSFORMATION

BACKGROUND THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Program Evaluations

Level 1: Reaction

Level 2: Learning Level 3: Application

Level 4: Impact

Level 5: ROI

INSPIRATION | INNOVATION | TRANSFORMATION

BACKGROUND THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Level 1 Evaluation Reaction Measures participants satisfaction with the program or process Examples: • • • •

Content was relevant to my work Represented an excellent use of my time Facilitator/faculty was knowledgeable The Learning environment was appropriate

INSPIRATION | INNOVATION | TRANSFORMATION

BACKGROUND THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Level 2 Evaluation Learning Measures changes in knowledge, skills, and attitudes Examples: • • • •

Identify the 4 features of... Demonstrate the use of... Name the 7 pillars of... Score at least 8 out of 10 on...

INSPIRATION | INNOVATION | TRANSFORMATION

BACKGROUND THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Level 3 Evaluation Application  Measures changes in performance or action  Needs management participation Examples: Self report, observation of practice and chart review

• 90% of users will follow the correct sequence of... • Within 3 months of the program, learners will demonstrate…

INSPIRATION | INNOVATION | TRANSFORMATION

BACKGROUND THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Level 4 Evaluation Business Impact  Measures changes in key business measures  Use existing measures Examples: • After 6 months, safety reports will show a 20% reduction in... • After 3 months, patients survey will report a... • Hospital acquired pressure ulcers will be by…

INSPIRATION | INNOVATION | TRANSFORMATION

BACKGROUND THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Level 5 Evaluation ROI  Compares the program benefits to the costs Examples: • ROI calculation is used to measure the monetary benefits to the costs • 1 year after implementation of the performance improvement project, a 2:1 benefit to cost ratio was achieved

INSPIRATION | INNOVATION | TRANSFORMATION

BACKGROUND THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

ROI Formula Program Benefits Benefits/Cost = Program Costs

Net program Benefits ROI = x 100 Program Costs

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

How do you select programs for higher level evaluations ? – – – – –

Linkage to strategic goals Stakeholders interest Cost of the program Visibility of the program Size of target audience

INSPIRATION | INNOVATION | TRANSFORMATION

SELF REPORT THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Level 3 Evaluations: Self Report • Methods • Interviews • Questionnaires/online survey (cost effective, standardized) • Data can be qualitative or quantitative

• Advantages • Learner’s own perspective • Can obtain data where observation is not possible

• Disadvantages • Potential validity problems

INSPIRATION | INNOVATION | TRANSFORMATION

SELF REPORT THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Timing of follow-up: When will the learner have the opportunity to apply the knowledge?

Convenience and constraints on collection ? – Level 3 – 1 week to 3 months

– 21 day rule

INSPIRATION | INNOVATION | TRANSFORMATION

SELF REPORT THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Self-Report Questionnaires/Surveys: Increasing the response rate – Clearly communicate the reason for the questionnaire – Keep it simple – Follow up reminders – Provide an incentive – Review the questionnaire at the end of the formal session

INSPIRATION | INNOVATION | TRANSFORMATION

OBSERVATION OF PRACTICE THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Level 3 Evaluation: Observation of Practice

Methods: – Observation of practice – Documentation review – Simulations

 Involve managers and staff in designing the tool  Specific template or checklist  All observers should use the same criteria and be objective

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Small-Group Case Study • Please divide into groups of 5 people

• Roles – 1 Director of Nursing Professional Development

– 4 Professional Development Specialists

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Case Study: Smith Medical Center This CNE program_______ has been running 4 times a year for the past 3 years. Many of our staff nurses have attended. They all love it! What difference is the program making in nursing practice on the patient care unit ? Are the nurses applying the content of the program?

INSPIRATION | INNOVATION | TRANSFORMATION

SELF REPORT THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Examples of Self Report Please describe a recent situation where your practice changed based on attending the "Staying Safe: How to Manage the Spectrum of Disruptive Behavior in the Workplace“ Please think back over the last several months since you completed Conflict Resolution and Management and describe a situation were you were able to apply the content from the course.

Please describe how applying the strategies learned in Conflict Resolution and Management impacted your nursing practice and patient care.

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Arrhythmia Knowledge Research Study • Purpose o The purpose of this study was to evaluate knowledge retention over time and clinical application of basic arrhythmia knowledge using simulation following exposure to an orientation program on basic arrhythmias.

• Method o Used simulation to measure application and knowledge-retention 3 months after a program.

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Research Questions • Is there a difference in pretest / posttest scores related to cardiac arrhythmia knowledge on a basic arrhythmia test (multiple choice, anatomy, rhythm strips) following exposure to a program on arrhythmias during central hospital orientation?

• Is there a difference in arrhythmia posttest scores on a basic arrhythmia test and retention of similar arrhythmias using scores from a simulated arrhythmia experience? • Is there a relationship between achievement on pretest / posttest basic arrhythmia test scores and nurse identified learning preferences?

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Sample • Inclusion criteria – A convenience sample of RNs obtained during an 18 month period of central hospital orientation .

• Exclusion criteria – Agency nurses, pediatric and non-MGH (Massachusetts General Hospital) nurses were excluded from the sample.

• Sample size determined using general power analysis (G*Power) – Sample size of 125 subjects provides 92% power to detect a moderate effect (0.15) with a significance of 0.05. – To control for 25% attrition rate a convenience sample of 138 RNs were recruited.

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Sample Size • Total enrollment (N = 138) • Pretest / Posttest Completion (N = 102) • Pretest / Posttest and Simulation (N = 62) Male Female

6.5% 93.5%

Age Range

21 – 66 yrs

Mean Age

29.5 yrs

Standard Deviation 9.09

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Process • Obtained Partners Human Research Committee IRB (Internal Review Board) approval. • Oral information and distribution of study fact sheet presented by principal and co-investigators (during central department orientation). • Demographic survey completed by participants who consented to the study. • Basic arrhythmia program (four hours) offered on the second day of nursing orientation to all nurses in this group. • Participants completed a written: – pretest one day before arrhythmia program – posttest (one hour) four weeks after arrhythmia program – posttest on identification of simulated rhythms (20 minutes) three months after original arrhythmia program

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Instrumentation • Demographic information sheet • Pretest / Posttest – – – – –

30 items Multiple choice Heart rate / interval calculation Fill-in the blank Rhythm identification

• Complete simulation scenarios answer sheet : – Eight rhythms simulated using a GE Healthcare Dash 4000 Monitor and Armstrong Medical Patient Simulator – Scoring (correct / incorrect) on eight identified simulated rhythms

• Post arrhythmia study survey / comments

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Simulation Scenario

Not to be produced without permission of the authors.

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Research Question #2 Is there a difference in arrhythmia posttest scores on a basic arrhythmia test and retention of similar arrhythmias using scores from a simulated arrhythmia experience? Simulation testing: • Data showed knowledge retention and clinical application in rhythm identification between posttest scores and clinical application in simulation testing at three months. Mean

Standard Deviation

n

Pretest

65.30

17.31

62

Posttest

87.59

12.08

62

Simulation

87.50

13.00

62

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Arrhythmia Study Conclusions – Significant difference in pretest / posttest achievement scores – Knowledge retention of arrhythmia content at three months post basic arrhythmia program – Simulation testing reinforced clinical application at three months post basic arrhythmia program Study findings contribute to the body of nursing knowledge and evaluation of clinical practice needed to insure safe, cost effective, patient centric, high quality patient care.

INSPIRATION | INNOVATION | TRANSFORMATION

OBSERVATION OF PRACTICE THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Example of Observation of Practice:

Conflict Engagement in Complex Systems

INSPIRATION | INNOVATION | TRANSFORMATION

OBSERVATION OF PRACTICE THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Example of Observation of Practice:

Improved Wound Healing Level 3 Outcome Evaluation

INSPIRATION | INNOVATION | TRANSFORMATION

OBSERVATION OF PRACTICE THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

• Accurately describe the characteristics of the patient’s wound? Yes □ No □ (i.e. wound bed, percentage of tissue types in wound, peri-wound area, etc.) • Correctly describe contributing factors to the wound/condition? Yes □ No □ (i.e. Pressure, friction, moisture, infection, inflow vs outflow disease, diabetic ulcer, etc) • If it is a pressure ulcer, correctly stage the pressure ulcer? Yes □ No □ N/A □ (i.e. Stage Pressure Ulcer: I, II, III, IV, unstageable)

• Develop an appropriate treatment plan and discuss the rationale for the plan? Yes □ No □ (Debride, add moisture, absorb moisture, anti-microbial, etc) • Verbalize the elements of documentation of the plan of care? Yes □ No □ (Including wound measurement, tissue type, peri-wound area, cleansing, peri-wound protection, wound bed, primary and secondary dressing, parameters for changing dressing, positioning, nutrition, etc.) • Discuss potential consults ? Yes □ No □

INSPIRATION | INNOVATION | TRANSFORMATION

OBSERVATION OF PRACTICE THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Improved Wound Healing Level 3 Outcome Evaluation • Lessons learned: – Pilot the observation tool first – Assistance of expert valid and reliable questions – Self report confidence tool

INSPIRATION | INNOVATION | TRANSFORMATION

OBSERVATION OF PRACTICE THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Example of Observation of Practice:

I-PASS

INSPIRATION | INNOVATION | TRANSFORMATION

OBSERVATION OF PRACTICE THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

The I-PASS Mnemonic I Illness Severity

Use standard language

P Patient Summary

Pertinent patient information, prompts may be different for each type of handoff

A Action List

To do list; timeline and ownership. What needs follow-up

S Situation Awareness & Contingency Planning Red flag issues, things to be aware of

S Synthesis by Receiver

Receiver summarizes what was heard, asks questions; restates key action / to do items

INSPIRATION | INNOVATION | TRANSFORMATION

OBSERVATION OF PRACTICE THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

I-PASS On-Unit Observation Report Scale (1 = not reported at all; 5 = reported comprehensively)

Pre-Educational Intervention (n=17)

Post-Educational Intervention (n=15)

4.00

3.50

3.00

2.50

2.00

1.50

1.00

0.50

-

Illness Severity

Patient Summary

Action List

Situation Awareness/ Synthesis by Receiver Contingency Planning

Mnemonic Components

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Summary What are the 5 levels of evaluation ? Why should the Professional Development Specialist gain expertise in the design of level 3 evaluations ?

What are the differences between self report and observation of practice ?

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

Application

How will you apply this content in your workplace ?

INSPIRATION | INNOVATION | TRANSFORMATION

THE NORMAN KNIGHT NURSING CENTER FOR CLINICAL & PROFESSIONAL DEVELOPMENT

References Avillion, Adrianne, Evidence Based Staff Development: Strategies to Create, Measure, and Refine Your Program, HCPro, Inc 2007 Kirkpatrick and Kirkpatrick, Creating a Post-training Evaluation Plan, Learning and Development, June 2013,p 2628 Nursing Professional Development Scope and Standards of Practice. American Nurses Association, 2010 Phillips, Patricia Pulliam. The Bottom-line on ROI. HRDQ Press, 2012 Phillips, Patricia Pulliam; Phillips, Jack J. 10 Steps to Successful Business Alignment. Alexandria: ASTD and ROI Institute, Inc., 2012 Phillips, Patricia Pulliam (ed.). Measuring and Evaluating Training. Alexandria: American Society for Training and Development, 2010 2014 MAGNET Application Manual, ANCC Sumner,L, Burke, S,.Chang,L.,McAdams, M., and Jones. D., Evaluation of Basic Arrhythmia Knowledge Retention and Clinical Application by Registered Nurses Journal of Nurses in Staff Development, Volume 28 number 2 (2012) Starmer AJ, Spector ND, Srivastava R, Allen AD, Landrigan CP, and Sectish TC. I-PASS, a Mnemonic to Standardize Verbal Handoffs. Pediatrics 2012; 129(2): 201-204.