Evaluation of a Critical Appraisal Program for Clinical Nurses: A Controlled Before-and-After Study Shu-Chen Chang, RN, MSN, Chin-Yi Huang, MHA, Shu-Yu Chen, RN, MSN, Yi-Cheng Liao, MD, Ching-Hsiung Lin, MD, PhD, and Hsiu-Hung Wang, RN, PhD, FAAN

abstract Background: Evidence-based practice is important to clinical health care professionals. Clinical nurses can make informed decisions by applying the best evidence to their care. However, standardized curricula on evidence-based medicine are lacking in nursing education programs. This study evaluated a critical appraisal education program and assessed its value in increasing nurses’ knowledge of critical appraisal and confidence in their critical appraisal skills. Methods: A controlled before-and-after study design was used. The education program integrated lectures, practice, and group discussion. A questionnaire was used to evaluate participants’ knowledge and confidence in critical appraisal pre- and postintervention. Results: Participants’ knowledge and confidence in critical appraisal improved significantly postintervention (both p < .001). Conclusion: A 1-day, small group discussion education program can effectively improve nurses’ knowledge and confidence in critical appraisal. Educators and administrators may replicate this education program to improve the quality of nursing care. J Contin Educ Nurs 2013;44(X):xx-xx.

C

ritical appraisal of the relevant published literature is an essential competence for nurses and is regarded by medical professionals as the standard for health care (Bernardo, Matthews, Kaufmann, & Yang, 2008). It is believed that the practice of evidence-based medicine would help health professionals to improve the quality of health care. The program conducted by the National Health Research Institutes of Taiwan, which includes information, resource support, and multifaceted camThe Journal of Continuing Education in Nursing · Vol 44, No X, 2013

paigns, has had a positive effect, increasing knowledge and skills in evidence-based practice and accelerating the dissemination of evidence-based practice in regional hospitals in Taiwan (Chiu, Weng, Lo, Shih, et al., 2010). Many health care institutions in Taiwan have recognized the importance of evidence-based practice and have implemented information systems to support physicians who practice evidence-based medicine (Hung, Ku, & Chien, 2012). However, Chiu, Weng, Lo, Shih, et al. (2010) found that many nurses reported that less support by health care administrators would be a barrier to their access to educational programs on evidence-based practice. Ms. Chang is Director, Department of Nursing, Changhua Christian Hospital, Changhua; and PhD candidate, College of Nursing, Kaohsiung Medical University, Kaohsiung, Taiwan. Ms. Huang is Administrator, Center for Evidence-Based Medicine, Changhua Christian Hospital, Changhua, Taiwan. Ms. Chen is Supervisor, Department of Nursing, Changhua Christian Hospital, Changhua; and doctoral student, Department of Nursing, National Taiwan University, Taipei, Taiwan. Dr. Liao is Director, Center for Evidence-Based Medicine, Changhua Christian Hospital, Changhua, Taiwan. Dr. Lin is Director, Division of Chest Medicine, Department of Internal Medicine, Changhua Christian Hospital, Changhua; Assistant Professor, Department of Respiratory Care, College of Health Sciences, Chang Christian University, Tainan; and Lecturer, School of Medicine, Chung Shan Medical University, Taichung, Taiwan. Dr. Wang is Professor, College of Nursing, Kaohsiung Medical University, Kaohsiung, Taiwan. The authors have disclosed no potential conflicts of interest, financial or otherwise. Address correspondence to Hsiu-Hung Wang, RN, PhD, FAAN, College of Nursing, Kaohsiung Medical University, No. 100, ShihChuan 1st Rd., Kaohsiung City, Taiwan. E-mail: [email protected]. tw. Received: April 15, 2012; Accepted: October 10, 2012; Posted: November 8, 2012. doi:10.3928/00220124-20121101-51

1

Evidence-based practice is highly recommended in the nursing literature. However, many nurses lack confidence in their own appraisal skills to evaluate evidence and determine what could be implemented in practice, mainly because of the lack of evidence-based nursing training in clinical practice (Tagney & Haines, 2009). Chiu, Weng, Lo, Hsu, et al. (2010) found that significant discrepancies exist between physicians and nurses in their awareness of, attitude toward, knowledge of, skills in, behavior toward, and barriers regarding evidence-based practice in Taiwan (Chiu, Weng, Lo, Hsu, et al., 2010). Even nurses who underwent a research course might still lack skills in appraising the value of research (Duffy, 2005). Therefore, developing objective and structured guidelines for critical appraisal is important to influence clinical decision making. By developing such appraisal skills, nurses can access and evaluate the most up-to-date resources and make appropriate clinical decisions according to the validity of the research. Skills and knowledge in critical appraisal are important aspects of implementing evidence into clinical practice (Milne, Krishnasamy, Johnston, & Aranda, 2007). Inconsistencies have been found in the amount and quality of different teaching approaches, and the lack of evidence supporting the approaches affects the usefulness of such training (Harewood & Hendrick, 2010). Therefore, well-designed continuing education programs for the appraisal and use of research are needed to improve nurses’ knowledge and skills and enhance their use of evidence in clinical practice. This study was undertaken to develop and conduct a critical appraisal education program for nurses, based on social learning theory, and to assess the improvement in nurses’ knowledge and confidence in critical appraisal after completion of the education program. METHODS This study and the questionnaires used were approved by the Evidence-Based Practice Committee of Changhua Christian Hospital, Changhua, Taiwan. Participants were recruited voluntarily after the research was announced. Before the program was started, the content and purpose of the pre- and postintervention were described to participants. To protect the rights of participants, the pre- and postintervention questionnaires were processed anonymously and no personal information was collected. Research Setting and Study Design This study was conducted in a 1,676-bed medical center located in central Taiwan. A controlled before-andafter study design was used. The participants were clini2

cal nurses at all levels of the clinical nursing ladder who were interested in evidence-based nursing. The clinical nursing ladder system refers to a hierarchical structure proposed by the Taiwan Nurses Association that divides nurses into four levels according to their clinical abilities and proficiency (Leu, Liao, Chang, & Su, 2009). A description of the program was distributed by the Evidence-Based Nursing Working Team. The head nurse of every nursing unit recommended a voluntary participant for the program. The difference in the total score of knowledge test of matched pairs (pre- to postintervention) is normally distributed. If the true difference in the mean response of matched pairs is 1.5, with a standard deviation of 2.0, then at least 36 subjects must be included in the study. The type II error (power) is 0.85 and the type I error probability is 0.05. To allow for a 10% dropout rate, the actual minimal sample size is 40. A total of 49 participants were enrolled in the study, and all 49 completed the education program. Participants’ knowledge of critical appraisal and confidence in their appraisal skills were evaluated pre- and postintervention. Educational Intervention The critical appraisal educational program consisted of three sessions. Each session included three parts and lasted for 2.5 hours. This 1-day education program was designed by the Evidence-Based Nursing Working Team of Changhua Christian Hospital and included a small group discussion. The participants were assigned to read three papers and other materials that were distributed before the program. They were also expected to be fully prepared to take part in the small group discussions. First, the lecturer introduced the topic for that session. All lectures were delivered by the same speaker, a physician who had experience in evidence-based practice and was qualified by the Taiwan Evidence-Based Medicine Association. Each lecture was followed by a small group discussion to reinforce the information. A tutor, who helped guide the small group discussion, was assigned to each group by the Evidence-Based Nursing Working Team. Each group was assigned to read three different types of clinical research papers: treatment studies, systematic reviews, and clinical practice guidelines. After the participants read and appraised an article by themselves, each group held a 50-minute discussion using the critical appraisal users’ guidelines and an appraisal instrument. Participants shared their appraisals of the paper with the other members of the small group. Finally, in a 40-minute session, the lecturer assisted the participants in reaching a consensus on the format of a larger group review of the critical appraisal users’ guideCopyright © SLACK Incorporated

lines and the appraisal instrument for the guidelines. Before the first session, participants were asked to complete the preintervention evidence-based nursing questionnaire on critical appraisal. They were also asked to complete the immediate postintervention questionnaire at the end of the education program. Instruments An evidence-based nursing questionnaire was developed by the research team, based on the literature (Cullum, 2000, 2001; Guyatt & Rennie, 2008) and the critical appraisal users’ guidelines to assess the effectiveness of the education program, including participants’ knowledge of critical appraisal and their perceived confidence in appraising evidence. The questionnaire included three major sections: (1) basic demographic information; (2) evaluation of basic knowledge of critical appraisal; and (3) self-evaluation of confidence in appraisal (appraisal of literature on treatment and intervention, systematic reviews, and clinical practice guidelines). The knowledge scale consisted of 8 multiple-choice questions, such as “Please arrange the levels of evidence of the results of ‘treatment’ studies from high to low”; and 12 true-or-false questions, such as “If the 95% confidence interval of relative risk includes 0, it indicates that the treatment results are not statistically significant” and “The number needed to treat refers to the number of patients who must be treated to prevent one adverse result (i.e., to promote good results).” The confidence scale consisted of 27 statements, such as “Is the research literature randomized?” and “This systematic review stated a specific clinical question clearly.” In knowledge questions, correct responses were scored as 1 and incorrect responses were scored as 0. The knowledge scores were then summed and ranged from 0 to 20. Internal consistency reliability, measured by KuderRichardson Formula 20, was 0.5 for dichotomous choice items. Confidence statements were scored on a five-point Likert scale, with a score of 5 representing “strongly confident,” or a positive confidence statement, and a score of 1 representing “strongly not confident,” or a negative confidence statement. The number of confidence statements was 9 (scored 9 to 45), 10 (scored 10 to 50), and 8 (scored 8 to 40). Cronbach’s alpha coefficient values of the scales were 0.94, 0.95, and 0.95. RESULTS All participants were female. The characteristics of the 49 study participants are shown in Table 1. Table 2 summarizes and compares the differences in the scores of the pre- and postintervention questionnaires for the The Journal of Continuing Education in Nursing · Vol 44, No X, 2013

TABLE 1

DEMOGRAPHIC CHARACTERISTICS OF THE PARTICIPANTS (N = 49) Characteristic

n (%)

Age (years) 26 to 30

12 (24.5)

31 to 35

26 (53.1)

36 to 40

5 (10.2)

> 40

6 (12.2)

Education Postgraduate

2 (4.0)

Undergraduate

38 (77.6)

College or less

9 (18.4)

Motivation for attending programa Interested

8 (16.3)

Needed for work

11 (22.4)

Recommended

29 (59.2)

Clinical nursing ladderb Level 1

1 (2.0)

Level 2

13 (26.5)

Level 3

32 (65.3)

Level 4

2 (4.1)

a

Note. One participant had another motivation and selected none of the three choices. bOne participant had a missing response (did not record) on the item regarding the clinical nursing ladder.

49 participants, including four sections: knowledge test, confidence in appraisal of the treatment study, confidence in appraisal of the systematic review, and confidence in appraisal of the clinical practice guidelines. Postintervention scores for all sections were increased compared with preintervention scores (all p values < .001) (Table 2). Before the education program, the participants achieved a mean score of 14.0 (SD = 2.0). The postintervention mean score increased to 15.8 (SD = 1.4), showing a significant improvement in participants’ knowledge of appraisal and confidence in their appraisal skills. On the postintervention questionnaire, participants rated their confidence in appraisal of the treatment study as moderate (M = 32.4, SD = 5.8) and rated their confidence in appraisal of the systematic review (M = 33.9, SD = 6.6) and the clinical practice guidelines (M = 27.1, SD = 4.8) as lower. Participants’ motivations for attending the appraisal education program appeared to affect the pre- and postprogram results. Repeated measure analysis of variance 3

TABLE 2

DIFFERENCES IN SUBCATEGORY SCORES BETWEEN PRE- AND POSTINTERVENTION EVIDENCE-BASED NURSING QUESTIONNAIRES (N = 49) Evidence-Based Nursing Questionnaire Total score of knowledge test, M ± SD Mean change from pre- to postintervention (95% CI) Confidence in appraisal of treatment study, M ± SD Mean change from pre- to postintervention (95% CI) Confidence in appraisal of systematic review, M ± SD Mean change from pre- to postintervention (95% CI) Confidence in appraisal of clinical practice guidelines, M ± SD Mean change from pre- to postprogram (95% CI)

Preprogram

Postprogram

pa

14.0 ± 2.0

15.8 ± 1.4

< .001

32.4 ± 5.8

< .001

33.9 ± 6.6

< .001

27.1 ± 4.8

< .001

1.8 (1.2 to 2.4) 24.1 ± 8.6 8.3 (6.0 to 10.6) 23.5 ± 8.8 10.4 (8.1 to 12.7) 18.1 ± 7.9 9.0 (6.7 to 11.3)

Note. CI = confidence interval. ap values were derived via paired t test of confidence in appraisal of the treatment study and confidence in appraisal of the systematic review; Wilcoxon signed-rank test was used for total scores of knowledge and confidence of appraisal of the clinical practice guidelines as a result of abnormal distribution.

ever, no significant effects were found on confidence in appraisal skills by the types of motivation. Participants’ age, education, and level on the clinical nursing ladder did not influence their mean knowledge or confidence scores.

Figure. Pre- to postintervention change in total score of knowledge test according to motivation. Data are shown as a bar chart showing M ± SD. Differences between pre- and postintervention scores for each type of motivation were compared with the Wilcoxon signed-rank test. ***p < .001, indicating significant differences between pre- and postintervention scores. Total score on the knowledge test was evaluated by participants’ motives—interested: 15.8 ± 1.8 vs. 16.1 ± 1.9, p = .317; needed for work: 14.4 ± 1.4 vs. 15.5 ± 1.6, p = .136; recommended: 13.4 ± 2.0 vs. 15.8 ± 1.2, p < .001 (n = 48; one study subject with another motivation was not included).

shows that the total score on the knowledge test might be associated with participants’ motivation for attending the education program. The changes in the total scores on the knowledge test from pre- to postintervention according to motivation are shown in the Figure. The participants who were recommended to attend the education program showed a significant improvement (preintervention 13.4 ± 2.0 vs. postintervention 15.8 ± 1.2, p < .001) in their knowledge of critical appraisal. How4

DISCUSSION Nurses who had been recommended to take the education program showed greater improvement in knowledge and confidence scores than those who took the education program because of their own interest in appraisal or because they believed that they needed appraisal skills for work. Those recommended to take the education program may have been given the responsibility to implement evidence-based nursing in their wards or to serve as tutors in hospital-led education programs, and this may have motivated them to expend greater effort and to learn more effectively during the program. The intent of the appraisal education program reported in this study was to increase nurses’ knowledge of critical appraisal as well as to increase their confidence in their appraisal skills. The most effective appraisal education programs integrate lectures, practice, and small group discussion, which enhance the appraisal competencies of nurses (Bernardo et al., 2008; Bradley et al., 2005; Brancato, 2006). Therefore, this education program included lectures, practice sessions, and small group discussion that focused on critical appraisal of treatment studies, systematic reviews, and clinical practice guidelines. A review of the literature found that perceived lack of time is the main barrier for nurses to access information and apply evidence-based approaches in their clinical Copyright © SLACK Incorporated

care (Bertulis, 2008). Other barriers include insufficient computer skills to search for studies, poor search and appraisal skills, poor understanding of statistics, and high pressure on nursing staff in most clinical settings (Bertulis, 2008). Gerrish, Ashworth, Lacey, and Bailey (2008) also reported barriers to access to evidencebased information, such as nurses’ lack of confidence and lack of authority. Tagney and Haines (2009) stated that, to ensure that quality evidence-based practice is implemented into clinical nursing care, a “realistic and practical structure” must be applied. Implementing a realistic research framework into clinical nursing practice may lead to a more active research culture (Tagney & Haines, 2009). Continuing education remains an accepted and effective way to promote evidence-based practice. Problem-based learning and models based on social learning theory have been effective in transferring knowledge to health care personnel. Social learning theory increases learners’ incentives by applying interactive lectures, clear educational goals, stepwise educational programs, interactive group work, and feedback (Bradley et al., 2005). Results of a 1-day workshop showed that a short-term intervention could enhance school nurses’ appraisal skills (Bernardo et al., 2008). Compared with other programs, the duration of the program reported in this study was short. Nevertheless, the results suggest that the benefits of the education program could be achieved in the short term, which is important for busy clinical nurses who report that they have inadequate time to attend a long-term education program. After the education program, participants responded that most of the sessions met or exceeded their expectations and had improved their abilities. In addition, the participants reported that the critical appraisal users’ guidelines provided a convenient tool to increase their awareness of appraisal. These guidelines, which are based on research and include an appraisal checklist, can help participants to appraise different aspects of research. Although nurses did not completely understand all of the research terminology, their pre- and postintervention scores indicated improved knowledge and improved confidence in their critical appraisal skills. Clearly, if clinical nurses are willing to apply critical appraisal of evidence-based medicine in their daily clinical practice, more appraisal-related programs must be developed to enhance their competency in this respect. One limitation of this study was that the research design contained only a single group and did not include random assignment of participants. This would limit the applicability of the study outcomes. The other The Journal of Continuing Education in Nursing · Vol 44, No X, 2013

key points Critical Appraisal Program

Chang, S.-C., Huang, C.-Y., Chen, S.-Y., Liao, Y.-C., Lin, C.-H., Wang, H.-H. (2013). Evaluation of a Critical Appraisal Program for Clinical Nurses: A Controlled Before-and-After Study. The Journal of Continuing Education in Nursing, 44(X), xxx-xxx.

1

Critical appraisal of relevant published literature is an essential competence for nurses and is regarded as the standard of health care by medical professionals.

2

A well-designed critical appraisal education program combines lectures, practice sessions, and group discussions focused on critical appraisal of treatment studies, systematic reviews, and clinical practice guidelines.

3

A 1-day, small-group discussion education program can effectively improve nurses’ knowledge of critical appraisal and confidence in their critical appraisal skills.

limitation was that only short-term study outcomes were evaluated, so the long-terms effects would not be validated. A more comprehensive evidence-based practice education program, including 3- to 6-month and even 1-year follow-up, should be designed to examine the long-term effects of the education program. CONCLUSION A 1-day education program was an efficient and effective way to improve nurses’ knowledge of critical appraisal and confidence in their critical appraisal skills. Education programs may help and encourage nurses to engage in evidence-based nursing practice and help clinical professionals to overcome commonly reported barriers to applying evidence-based practice. Administrators can provide evidence-based practice education programs specifically designed for nurses instead of those designed for physicians. A comprehensive education program can be included in the advanced curriculum of continuing nursing education. Clinicians who have critical appraisal skills will be able to determine the importance of research studies and improve their clinical decision making. REFERENCES Bernardo, L. M., Matthews, J. T., Kaufmann, J. A., & Yang, K. (2008). Promoting critical appraisal of the research literature: A workshop for school nurses. The Journal of Continuing Education in Nursing, 39(10), 461-467. Bertulis, R. (2008). Barriers to accessing evidence-based information. Nursing Standard, 22(36), 35-39. Bradley, P., Nordheim, L., Innvær, S., & Thompson, C. (2005). A systematic review of qualitative literature on educational interventions

5

for evidence-based practice. Learning in Health & Social Care, 4(2), 89-109. Brancato, V. C. (2006). An innovative clinical practicum to teach evidence-based practice. Nurse Educator, 31(5), 195-199. Chiu, Y. W., Weng, Y. H., Lo, H. L., Hsu, C. C., Shih, Y. H., & Kuo, K. N. (2010). Comparison of evidence-based practice between physicians and nurses: A national survey of regional hospitals in Taiwan. Journal of Continuing Education for Health Professionals, 30(2), 132-138. Chiu, Y. W., Weng, Y. H., Lo, H. L., Shih, Y. H., Hsu, C. C., & Kuo, K. N. (2010). Impact of a nationwide outreach program on the diffusion of evidence-based practice in Taiwan. International Journal of Quality Health Care, 22(5), 430-436. Cullum, N. (2000). Evaluation of studies of treatment or prevention interventions. Evidence-Based Nursing, 3(4),100-102. Cullum, N. (2001). Evaluation of studies of treatment or prevention interventions: Part 2. Applying the results of studies to your patients. Evidence-Based Nursing, 4(1), 7-8. Duffy, J. R. (2005). Critically appraising quantitative research. Nursing and Health Sciences, 7(4), 281-283. Gerrish, K., Ashworth, P., Lacey, A., & Bailey, J. (2008). Developing

6

evidence-based practice: Experiences of senior and junior clinical nurses. Journal of Advanced Nursing, 62, 62-73. Guyatt, G., & Rennie, D. (2008). Users’ guides to the medical literature (2nd ed.). New York, NY: McGraw-Hill. Harewood, G. C., & Hendrick, L. M. (2010). Prospective, controlled assessment of the impact of formal evidence-based medicine teaching workshop on ability to appraise the medical literature. Irish Journal of Medical Science, 179(1), 91-94. Hung, S. Y., Ku, Y. C., & Chien, J. C. (2012). Understanding physicians’ acceptance of the Medline system for practicing evidencebased medicine: A decomposed TPB model. International Journal of Medical Informatics, 81(2), 130-142. Leu, L. J., Liao, H. C., Chang, I. C., & Su, Z. Y. (2009). Applying nonsynchronized e-learning to the nursing clinical ladder system. Journal of Medical Systems, 34(5), 909-917. doi:10.1007/s10916-009-9306-5 Milne, D. J., Krishnasamy, M., Johnston, L., & Aranda, S. (2007). Promoting evidence-based care through a clinical research fellowship programme. Journal of Clinical Nursing, 16(9), 1629-1639. Tagney, J., & Haines, C. (2009). Using evidence-based practice to address gaps in nursing knowledge. British Journal of Nursing, 18(8), 484-489.

Copyright © SLACK Incorporated