Review articles
A systematic review of the effectiveness of critical appraisal skills training for clinicians Rod Taylor,1 Barnaby Reeves,1 Paul Ewings,2 Sarah Binns,3 John Keast4 & Rebecca Mears1
1 Summary The aim of this paper is to undertake a descriptive systematic review of the effectiveness of critical appraisal skills training for clinicians. Of the 10 controlled studies which examined this issue and were found to meet the eligibility criteria of this review, all used a study population of either medical students or doctors in training. The studies used a variety of different intervention `dosages' and reported a range of outcomes. These included participants' knowledge of epidemiology/biostatistics, their attitudes towards medical literature, their ability to appraise medical literature, and medical literature reading behaviour. An overall improvement in assessed outcomes of 68% was reported after critical appraisal skills training, particularly in knowledge relating to epidemiology and biostatistics. This review appears to provide some
Introduction The philosophy of evidence-based practice continues to develop apace across many health care systems of the world.1 For clinicians to follow such an evidence-based approach they must be able to: (1) turn problems of their clinical practice into focused questions; (2) search in a focused way for published literature to address these questions; (3) appraise this literature critically for its usefulness and scienti®c validity; and (4) apply the results of this appraisal back to their practice.2 Although training in these skills is now a part of the syllabus of many undergraduate medical schools, nursing and other paramedical training establishments,
1
HSRU, Department of Public Health & Policy, London School of Hygiene & Tropical Medicine; 2Taunton & Somerset R & D Support Unit, Musgrove Park Hospital, Taunton; 3Department of Clinical Veterinary Science, University of Bristol; 4Research & Development Support Unit, Postgraduate Medical School, University of Exeter Correspondence: Dr Rod Taylor, Health Services Research Unit, Department of Public Health & Policy, London School of Hygiene & Tropical Medicine, Keppel Street, London WC1E 7HT, UK.
120
evidence of the bene®t of teaching critical appraisal skills to clinicians, in terms of both knowledge of methodological/statistical issues in clinical research and attitudes to medical literature. However, these ®ndings should be considered with caution as the methodological quality of studies was generally poor, with only one study employing a randomized controlled design. There is a need for educators within the ®eld of evidence-based health to consider the implications of this review. Keywords *Clinical competence; controlled clinical trials; health personnel, *education; programme evaluation. Medical Education 2000;34:120±125
many quali®ed medical practitioners still lack the expertise required.3 A number of approaches have been developed to help clinicians enhance their critical appraisal skills, such as the publication of the JAMA appraisal checklists,4 the BMJ series `How to read a paper',5 the publication of some key textbooks on evidence-based practice,2 and the development of a number of critical appraisal short courses and workshops. In the United Kingdom, there has been a substantial investment in critical appraisal skills training via the NHS Executive Of®ces, for example, the Critical Appraisal Skills programme (CASP) developed by the Public Health team in Oxford6 and the programme of North Thames Research Appraisal Group (NTRAG). The level of investment already committed to the development and roll-out of critical appraisal skills training raises a number of questions about effectiveness and cost effectiveness. Do such educational interventions change the knowledge base of attendees? Do they enhance the attitudes towards the use of evidence? and importantly, Do they increase the use of evidence in clinical decision making?
Ó Blackwell Science Ltd ME D I C A L ED U C A T I ON 2000;34:120±125
A review of critical appraisal skills training
·
R Taylor et al.
In 1993, Audet and colleagues published an article, in French, which reviewed the effect of teaching critical appraisal skills to medical students.7 From a search for English and French articles published between 1980 and 1990 indexed on MEDLINE and FAMLI, and a search of bibliographies, they identi®ed 10 studies to review. Of these 10 studies, they found that few demonstrated a statistically signi®cant improvement in knowledge of epidemiology and biostatistics, reading habits and the ability to critically appraise a scienti®c article. Moreover, they found the methodological quality of the studies to be generally poor, with only half of the studies reaching a quality score that the authors deemed satisfactory. Only one of the studies utilized a randomized controlled design.4 Given that this review of Audet et al. was limited to medical students, or newly quali®ed doctors, and the possibility that other studies of higher methodological quality may have been published subsequently, we undertook to update this review. We searched for studies, from 1966 to December 1997, that examined the effectiveness of critical appraisal skills training on any of the health care professional groups.
Methods
121
appraisal skills; (b) the presence of a comparison (control) group which received either no educational intervention or a `placebo' educational intervention; (c) a study population of health care professionals; and (d) an objective measurement of either an educational or health care outcome. Assessment of study quality
We adapted a protocol described previously to score the internal validity of each study.8 The protocol assessed ®ve dimensions: (a) the manner in which subjects were assigned to intervention or control groups; (b) control of selection bias after treatment allocation; (c) blinding of participants (to the purpose of the study) and investigators; (d) objectiveness of assessment of the outcome; and (e) adequacy of study power (see section below on data extraction). Overall score could range from 0 (worst possible) to 12 (best possible). The method used is available from the authors. Quality was independently assessed by two of the authors (RT and SB), with agreement evaluated by the weighted Kappa statistic. The weighted Kappa score for the two observers' assessment of quality score was 0á94 2 indicating very good agreement.9 Any disagreements were resolved by discussion.
Inclusion of studies
Data extraction
An initial group of studies evaluating the effectiveness of various critical appraisal teaching strategies was identi®ed from the previous review by Audet and colleagues. Additional English and non-English studies were identi®ed by a computerized literature search of the MEDLINE (1966 to December 1997), BIDS-ISI (1981 to December 1997), CINAHL (1982 to December 1997), EMBASE (1980 to December 1997) and EDINA BIOSIS (1985 to December 1997) databases. These searches used combinations of the key words: teaching, critical appraisal, medical education, evidence-based medicine and evidence-based health care. The System for Information on Grey Literature in Europe (SIGLE) was also searched using these terms. The bibliographies of the references obtained by these methods were also inspected. In addition, previous reviews, and major textbooks on medical education and evidence-based health care were searched for references.
Two authors (RT and SB) independently extracted the following data from each of the 10 studies included: (a) country of origin; (b) study population; (c) study setting; (d) details of study design; (e) description of the educational and control interventions; (f) outcome measures; and (g) study results. Study results were classi®ed as positive, negative or inconclusive. Positive studies were those that demonstrated a statistically signi®cant improvement in outcome at the level of p < 0á05. Negative studies showed either no improvement, despite suf®cient statistical power to do so, or a decrement in outcome. Inconclusive studies failed to demonstrate a change but lacked the statistical power to demonstrate or exclude an important difference (de®ned here as a standardized difference of 0á5, for 80% power and 5% signi®cance). Statistical pooling of the results by meta-analysis has not been undertaken because of the heterogeneity of study design, methods and outcomes.10
Eligibility criteria
We included studies in the systematic review if they met all four of the following criteria: (a) delivery of an educational intervention aimed at improving critical
Results Of the studies identi®ed by Audet et al. four failed to include a control group and were therefore excluded.
Ó Blackwell Science Ltd ME D I C AL ED U C AT I ON 2000;34:120±125
122
A review of critical appraisal skills training
·
R Taylor et al.
The combined search strategy identi®ed 247 potentially relevant abstracts (92 from MEDLINE, 83 from BIDSISI, 10 from EMBASE, 39 from EDINA BIOSIS and 23 from CINAHL). Fourteen studies (six from Audet et al. and eight new studies) were found to ful®l the eligibility criteria of this review.11±24 However, four were abstracts from conference proceedings and were excluded because, despite efforts to contact the authors, we could not obtain the necessary additional information.21±24 A total of 10 studies were therefore included in this review. The median quality assessment score for the 10 studies was only 3, with seven of the studies scoring 4 (out of a possible score of 12) or less. The studies included either medical students (n 6), or newly quali®ed physicians (n 4). The educational interventions ranged widely from a total of 180 min over a 1-week period to 16 h over the period of a year. The speci®c details of the educational interventions are presented in Table 1. The control groups received either no educational input (n 6), a general medical input (n 2), or a traditional epidemiological education (n 2). The 10 studies included the assessment of 22 separate outcomes, these outcomes being grouped into four broad areas: knowledge of epidemiology/statistics, attitudes towards medical literature, ability to appraise a research article, and medical literature reading behaviour (see Table 2). Fifteen (68%) outcomes were positive, 4 (18%) were negative and the remainder were inconclusive. All studies assessed knowledge, one study having three separate knowledge outcomes. In 10 out of these 12 instances (83%) a positive result was reported. Both studies that assessed attitudes towards evidence reported a positive result. Of the four assessments of critical appraisal ability, two reported negative results, one inconclusive and one positive. Two of the three studies that assessed reading behaviour reported a `positive result', the other being inconclusive. These positive changes in reading behaviour were de®ned in terms of either reading more journals15 or, reading more effectively.17 Studies with a control group receiving no educational input reported a higher proportion of positive outcomes (10 out of 13, 77%) than the studies with a comparative educational input (4 out of 9, 44%).
Discussion This review appears to provide some evidence that teaching critical appraisal skills improves both knowledge of methodological and statistical issues in clinical research and attitudes towards medical literature amongst participants. Evidence for an improvement in
the ability of participants to appraise evidence critically, or for a change in medical literature reading habits was not convincing. Even the evidence for an improvement in knowledge must be questioned. Most studies had poor internal validity, only one using a randomized controlled trial design and only two studies (one of which was the RCT) scoring more than 50% on the quality assessment. Studies of educational interventions are particularly vulnerable to confounding and bias. Participants, for example, are often self-selected, and given that the intervention is not blinded it may introduce a so-called `Hawthorne effect'; modi®cations in behaviour that might be stimulated by a `researcher'/educator purposefully observing actions. In view of the large investment being made in teaching critical appraisal skills to health care practitioners in the UK and other countries,25 it is important to consider the generalizability of the evidence which we reviewed. Its relevance to current critical appraisal skills programmes is dubious with respect to all aspects of the studies, i.e. the populations studied, the interventions evaluated and the outcomes used to assess effectiveness. In all studies, participants were either medical students or medical interns/residents. It might be argued that these groups are likely to be more receptive to new knowledge and better able to retain it than typical NHS practitioners working in a variety of professions. On the other hand, they may be less able than practitioners with years of experience of providing health care to interpret the generalizability of evidence, an important aspect of appraising evidence critically. The interventions employed by the authors appear to have been largely `classroom' based, with the majority involving eight or more sessions of an hour or more, representing a large investment of time. With the exception of diploma or degree courses, most programmes currently being used to teach critical appraisal skills to practitioners are of a much shorter duration ± typically one session of about three hours, with problem-based learning techniques being used to address applied clinical problems. Intuitively, it seems unlikely that senior practitioners, often those most in need of training in critical appraisal skills, would commit themselves to programmes that required them to attend multiple sessions. It is also unclear whether an improvement in knowledge of methodological and statistical issues in clinical research, and a more positive attitude towards evidence represent a suf®cient bene®t from critical appraisal programmes. It is widely recognized that a practitioner's knowledge of the effectiveness of an
Ó Blackwell Science Ltd ME D I C A L ED U C A T I ON 2000;34:120±125
A review of critical appraisal skills training
·
R Taylor et al.
123
Table 1 Summary of study origin, size, and description of educational intervention Author
Country
Intervention group
Control group
Intervention details
Gehlbach et al. 198010
USA
23 (2nd & 3rd year) residents
12 (1st year) residents No input
2 months (8 sessions) 1 h/session A combination of taught & practical sessions on critical appraisal of medical articles.
Riegelman et al. 198311
USA
150 (1st year) medical students
146 (4th year) medical students Biostatistics course
1 years (10 sessions) 1 h/session A combination of taught & practical sessions in appraisal of the clinical research articles.
Viniegra et al. 198612
Mexico
37 (1st & 2nd year) internal medical students
10 internal medical residents No input
13 sessions Discussion of methodology (design, analysis & validity of conclusions) of 25 published papers (effectiveness of therapy, diagnosis, natural history of disease).
Radack et al. 198613
USA
22 (4th year) medical students
12 (4th year) medical students No input
2 months (5 sessions) 50 mins/session Practical sessions requiring appraisal of the validity of an article with accompanying clinical scenario.
Riegelman, 198614
USA
140 (4th year) medical students
82 (4th year) medical students No input
1 years (16 sessions) 1 h/session 12 h of lectures on epidemiology and statistics followed by 4 h of seminars reviewing articles.
Bennett et al. 198715
Canada
49 (senior) medical students
43 (senior) medical students Received input from tutors not trained in critical appraisal.
8 weeks (8 sessions) 2 h/session Practical sessions requiring solution of clinical scenarios based on appraisal of treatment and diagnosis research articles.
Linzer et al. 198816
USA
22 medical interns
22 medical interns Lecture series on ambulatory medicine
9á5 months (5 sessions) 1 h/session Journal club which involved critique of a research article. A staff member facilitated & provided teaching for each session.
Kitchens et al. 198917
Canada
51 medical residents
32 medical residents Lectures on ambulatory medicine
8 weeks (8 sessions) 30±45 mins/session Practical sessions involving critique of a research article assisted by an accompanying reading.
Frasca et al. 199218
USA
48 (3rd year) medical students
44 (3rd year) medical students No input
10 weeks (10 sessions) 1á5 h/session Two taught sessions ± one on literature searching & one appraising articles. Eight sessions involved problem-based questions requiring application of skills.
Landry et al. 199419
USA
65 (3rd year) medical students
81 (3rd year) medical students No input
1 week (2 sessions) 90 mins/session Two taught sessions ± one on study design and the other on methods of appraisal of the medical literature, both supported by written material.
As described by the author.
intervention is not suf®cient to ensure that the intervention is used. Therefore, it also seems unlikely that improved knowledge of the technical issues that are involved in clinical research is suf®cient to ensure that
practitioners seek out, appraise and implement evidence appropriately. While we should evaluate the effectiveness of educational interventions, such as Critical Appraisal Skills
Ó Blackwell Science Ltd ME D I C AL ED U C AT I ON 2000;34:120±125
124
A review of critical appraisal skills training
·
R Taylor et al.
Table 2 Summary of study design, outcomes, results and methodological quality assessment
Author
Design
Main outcome(s)
Result
Quality assessment
Gehlbach et al. 1980
Prospective cohort Post assessment only
Knowledge of biostatistics & epidemiology
Positive
2
Riegelman et al. 1983
Cross-sectional
Attitude towards medical research evidence Reading behaviour Perceived appraisal knowledge Knowledge of epidemiology and statistics
(1) (2) (3) (4)
Positive Negative Negative Negative
2
Vinegra et al. 1986
Non-randomized trial Pre & post assessment
(1) Theoretical knowledge of clinical research (2) Applied knowledge of clinical research I: Global appreciate judgement scale Applied knowledge of clinical research II: Analytic reasoning
(1) Positive (2) Inconclusive (3) Positive
3
Radack et al. 1986
Prospective cohort Pre & post assessment
Knowledge of biostatistics & epidemiology
Inconclusive
1
Riegelman, 1986
Retrospective cohort Pre & post assessment
(1) Perceived appraisal competence (2) Knowledge of epidemiology and statistics (3) Reading behaviour
(1) Positive (2) Positive (3) Positive
2
Bennett et al. 1987
Non-randomised trial Pre & post assessment
Knowledge of biostatistics & epidemiology
Positive
5
Linzer et al. 1988
Randomized trial Pre & post assessment
(1) Perceived reading habits (2) Knowledge of epidemiology & statistics (3) Critical appraisal skill
(1) Positive (2) Positive (3) Inconclusive
7
Kitchens et al. 1989
Non-randomized trial Pre & post assessment
Knowledge of clinical epidemiology
Positive
2
Frasca et al. 1992
Prospective cohort Post assessment
(1) Knowledge of critical appraisal (2) Knowledge of literature searching
(1) Positive (2) Positive
4
Landry et al. 1994
Non-randomized trial Pre & post assessment
(1) Knowledge of epidemiology & statistics (2) Attitudes towards use of medical literature (3) Critical appraisal skills
(1) Positive (2) Positive (3) Negative
7
Training programmes, it is dif®cult to examine this effect without instruments that are suf®ciently sensitive to measure their true value. Further investment of time and resources is therefore required in order to develop instruments that can accurately measure the components that contribute to the effectiveness of such interventions.
References 1 Sackett D, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. Evidence-based medicine: what it is and what it isn't. BMJ 1996;313:169±71. 2 Sackett D, Richardson WS, Rosenberg WMC, Haynes RB. Evidence-based medicine: how to practice and teach EBM. London: BMJ Publishing Group; 1997. 3 General Medical Council, Tomorrow's Doctors. Education Committee of the General Medical Council in pursuance of 4 section 5 of the Medical Act 1983. London: GMC; December 1993. 3
4 Oxman AD, Sackett DL, Guyatt GH. For the Evidence Based Medicine Working Group. Users guides to the medical literature: I. How to get started. JAMA 1993;270:2093±5. 5 Greenhalgh T. How to read a paper. Getting your bearings (deciding what the paper is about). BMJ 1998;315(7102):243±6. 6 Milne R, Donald A, Chalmers L. Piloting short workshops in critical appraisal of review. Health Trends 1995;27:120±3. 7 Audet N, Gagnon R, Ladouceur R, Marcil M. L'enseignement de l'analyse critique des publications scienti®ques medicales est-il ef®cace? Revision des etudes et de leur qualite methodologique. Can Med Assoc J 1993;148:945±52. 8 Chalmers I, Adams M, Dickersin K, Hetherington J, TarrowMordi W, Mienert C, Tonascia S, Chalmers TC. A cohort study of summary reports of controlled trials. JAMA 1990;263:1401±5. 9 Landis R, Koch G. The measurement of observer agreement 5 for categorical data. Biometrics 1977;33:159±74.
Ó Blackwell Science Ltd ME D I C A L ED U C A T I ON 2000;34:120±125
A review of critical appraisal skills training
·
R Taylor et al.
10 Mulrow CD, Oxman AD, eds. Cochrane Collaboration Handbook (Updated 1 March 1997). In the Cochrane Library [database on disk & CD-ROM]. The Cochrane Collaboration. Update Software. 1996-updated software, 1996. 11 Gehlbach SH, Bobula JA, Dickinson JC. Teaching residents to read the medical literature. Med Educ 1980;55:362±5. 12 Riegelman RK, Pover GL, Ott JE. Medical student's skills, attitudes and behaviour needed for literature searching. Med Educ 1983;58:411±7. 13 Viniegra L, Leon SP, De Calva J, Leon P, Loria A, Lisker R, Vargas F, Herrara M. La capacidad de critica en investigacion clinica en un grupo residentes de medicina interna. La Rev Invest Clin (Mex) 1986;38:83±7. 14 Radack KL, Valanis B. Teaching critical appraisal and application of medical literature to clinical problem-solving. Med Educ 1986;61:329±31. 15 Riegelman RK. Effects of teaching ®rst-year students skills to read the medical literature. Med Educ 1986;61:454±60. 16 Bennett KJ, Sackett DL, Haynes RB, Neufeld VR, Tugwell P, Roberts R. A controlled trial of teaching critical appraisal of the clinical literature to medical students. JAMA 1987;257:2451±4. 17 Linzer M, Trig Brown J, Fazier LM, De Long ER, Siegel WC. Impact of a medical journal club on house-staff reading habits, knowledge, and critical appraisal skills. A randomised controlled trial. JAMA 1988;260:2537±41. 18 Kitchens JM, Pfeifer MP. Teaching residents to read medical literature: a controlled trail of a curriculum in
125
19
20
21
22
23 24
25
critical appraisal/clinical epidemiology. J Gen Intern Med 1989;4:384±7. Frasca MA, Dorsch JL, Aldag JC, Christensen RG. A multidisciplinary approach to information management and critical appraisal instruction: a controlled study. Bull Med Libraries Assoc 1992;80:23±8. Landry FJ, Pangaro L, Kroenke K, Lucey C, Herbers J. A controlled trial of a seminar to improve medical student attitudes towards, knowledge about, and use of the medical literature. J Gen Intern Med 1994;9:436±9. Hofmann JC, Taylor WC. Teaching critical appraisal of the medical literature: effects on attitudes of ®rst-year medical students. J Gen Intern Med 1997;12 (Suppl.):92. Ghali WA, Eskew AH, Gupta M, Hershman WY. Successful teaching in evidence-based medicine. J Gen Intern Med 1997;12 (Suppl. 1):91. Green ML, Ellis PJ. A controlled trial of an evidence-based medicine. J Gen Intern Med 1997;12 (Suppl. 1):91. Hayward R, Bass E, Grace D, Kean D, Howard D, Barker L. Learning critical appraisal in a medical house staff clinic. Clin Res 1990;38:728A. Buxton M, Hanney S. Evaluating the NHS research and development programme: will the programme give value for money? J R Soc Med. 1998;91 (Suppl. 35):2±6.
Received 7 May 1999; editorial comments to authors 14 July 1999; accepted for publication 26 July 1999
Ó Blackwell Science Ltd ME D I C AL ED U C AT I ON 2000;34:120±125
