The Effectiveness of Computer-Aided, SelfInstructional Programs in Dental Education: A Systematic Review of the Literature Harold Rosenberg, B.Sc.; Helen A. Grad, M.Sc.Phm.; David W. Matear, D.D.S., M.Sc. Abstract: Computer-aided learning (CAL), self-instructional programs provide an accessible, interactive, and flexible way of presenting curriculum material. In order to assess the effectiveness of CAL programs in dental education, a systematic review of the published literature comparing CAL with other teaching methods was performed. A systematic search of the published literature was performed. Articles formally assessed for inclusion had to meet the following criteria: randomized controlled trials comparing CAL with any other method of instruction, and the use of academically homogeneous dental students or dental professionals with objective, predefined outcome criteria measuring performance, time spent, and attitudes. The searches located a total of 1,042 articles; of these, only twenty-seven articles met the inclusion criteria. Further quality assessment identified twelve studies that were included in the final review. Five of the studies documented statistically significant differences in outcome measures (scores on multiple choice, written or oral tests, and clinical performance) favoring CAL over comparison group(s), while six revealed no statistically significant differences. One study documented a greater improvement in test scores in the seminar group over the CAL group. Participants’ attitudes towards CAL in the included studies are also discussed. Our study concluded that CAL is as effective as other methods of teaching and can be used as an adjunct to traditional education or as a means of self-instruction. Mr. Rosenberg is a Senior D.D.S. Student, Ms. Grad is a Senior Lecturer, Department of Pharmacology, and Dr. Matear is Associate Professor, Department of Community Dentistry and Dean of Clinical Affairs, all at the Faculty of Dentistry, University of Toronto. Direct correspondence and requests for reprints to Ms. Helen A. Grad, Faculty of Dentistry, University of Toronto, 124 Edward St., Room 252, Toronto, Ontario, Canada M5G1G8; 416-979-4909, ext. 4360 phone; 416-979-4936 fax; [email protected]. This work was funded by the Network for Oral Research Training and Health, CIHR. Key words: computer-assisted learning, computer-based instruction, dental education, self-study, systematic review Submitted for publication 12/3/02; accepted 3/3/03

S

elf-instruction has been shown to be an effective method of learning in dental education. A meta-analysis of self-instruction in dental education by Dacanay, integrating findings from thirtyfour comparative studies, showed that educators who individualize their classes could expect, on average, a small to moderate positive effect on achievement.1 Their conclusions were in accordance with a study by Williams where it was found that self-instruction was capable of increasing cognitive knowledge significantly in a shorter period of time and with greater student satisfaction over conventional methods.2 One such means of providing self-instruction is through computer-based instructional programs. Computer-based, self-instructional programs provide an accessible, interactive, and flexible way of giving multimedia presentations that utilize textual materials, visuals, sound, and motion. Since the development of the personal computer in 1981, computers have been used extensively in educational settings. Computer programs complement conventional teaching while providing a means for students to learn at their own pace. Computer-based instruction (CBI) in the health

524

profession, also known as Computer-Aided Learning (CAL) or Computer-Aided Instruction (CAI), is becoming a popular vehicle to provide information to students, patients, and practitioners alike. A meta-analysis of 254 controlled evaluation studies of Computer-Based Instruction (CBI) by Kulik and Kulik revealed a positive effect on students.3 The study revealed that CBI programs raised student examination scores by 0.3 standard deviations, produced positive changes in student attitudes toward teaching and computers, and decreased the amount of time needed for instruction. CAL in dentistry first emerged in 1971 with its introduction at the University of Kentucky.4 In the past two decades, the progress of communication technology and the wide use of the Internet have made CAL programs readily available, while their availability without time constraints makes them an attractive alternative to conventional learning. There are CAL programs on the market in various dental disciplines, including: oral and gross anatomy, orthodontics, endodontics, restorative dentistry, radiology, oral pathology, geriatrics, prosth-

Journal of Dental Education ■ Volume 67, Number 5

odontics, and periodontology. Several studies have evaluated the effectiveness of CAL versus other teaching methods in dentistry. Many of these studies, however, use the participant’s perception of his or her experience with a learning program as the only outcome measure for evaluating the effectiveness of the CAL program.5-10 In this review we sought to identify high-quality, well-designed, randomized controlled intervention studies that are available in the published literature comparing CAL to other teaching methods.

Objectives The objectives of this systematic review of the CAL literature were to: 1. Assess the effectiveness of computer-aided, selfinstructional programs in dental education, and 2. Review the published literature and develop evidence-based guidelines for the use of CAL in dental education.

Methods Only randomized controlled trials (RCTs) comparing computer-aided learning with any other method of instruction (self-instructional or tutoraided instruction) were considered. Only studies that included the following types of students were eligible for the review: academically homogeneous undergraduate dental students, graduate dental students, dentists, and faculty at dental schools. Studies considered for inclusion in the review had to use one or more of the following types of outcome measures: objectively measured post-test scores on multiple choice, written or oral tests; objectively measured performance on a clinical procedure or clinical interview; or objectively measured time spent on CAL programs to learn the material presented and responses to questionnaires dealing with the participant’s attitudes towards the mode of learning. Electronic searching by an information technology expert and non-electronic searching were used. Subject-specific search strategies were developed using vocabulary mapping (mp) which included MeSH terms, text words, and index terms gathered from existing collections of articles known to be relevant. Non-electronic searching was done based on cited studies in relevant papers.

May 2003

■

Journal of Dental Education

Our electronic search strategy consisted of the following: 1. MEDLINE 1966 to June week 3, 2002 using: self-study (mp), computer-assisted instruction (mp), computer aided-instruction (mp), computer-based instruction (mp), problem-based learning (mp), dental education/or education, dental (mp explode), medical education (mp) [mp=title, abstract, cas registry/ec number word, MeSH subject heading] 2. The Cochrane Library to the second quarter of 2002, including Cochrane Controlled Trials Register, Cochrane Database of Systematic Reviews, Cochrane Database of Abstracts of Reviews of Effectiveness using: self-study, computer-assisted instruction (mp), education dental (mp) [mp=title, original title, abstract, MeSH headings, heading words, keyword] 3. EMBASE (Excerpta Medica) 1980 to 2002 week 24 using terms as in MEDLINE 4. ERIC (Educational Resources Information Center) 1970 to June 2002 using: computer-assisted instruction, intelligent tutoring systems, courseware, learner-controlled instruction, and dental education 5. CINAHL (Cumulative Index to Nursing & Allied Health) 1982 to June week 4 2002 using: self-study, computer-assisted instruction, and education 6. LISA (Library and Information Science Abstracts) 1969 to May 2002 using terms as in CINAHL 7. Psycinfo (Psychological Information) 1970 to May 2002 using terms as in CINAHL 8. IPA (International Pharmaceutical Abstracts) 1970 to June 2002 using: computer-assisted instruction, self-study One reviewer, who excluded articles clearly of no relevance to the study, initially screened the results of all searches. This screening process eliminated any studies that were not dentistry-based, lacked a comparison group, and lacked the criteria previously mentioned for types of studies, participants, and outcomes. Copies of all remaining (not excluded) articles were retrieved and reviewed by two independent assessors who judged whether they were eligible for formal assessment according to the criteria stated above. The validity and strength of the selected studies were assessed using a checklist of nine criteria that measures the strength of an intervention study adapted by Dr Jim Leake (Table 1). The studies analyzed had to meet a minimum of seven

525

Table 1. Example of checklist used to assess the strength of educational intervention studies Author: Kay Was a strong design used to assess efficacy? Were outcomes validly and reliably measured? Were interventions validly and reliably measured? Was the assignment of subjects randomized? Were both pre-test and post-test values documented? If finding were of no difference was the power of the study >80%? Was loss of follow up less than 20% and balanced between test and controls? Were the groups similar at the start of the trial? Aside from the experimental intervention, were the groups treated equally? TOTAL

Yes Yes Yes Yes Yes Yes Yes ?* Yes 8/9

*If any of the items on the checklist were not clearly stated or easily identified in the article, a question mark was given for that category and not counted in the total score. Source: Checklist was adapted by Dr. Jim Leake from Fletcher et al., Clinical epidemiology: the essentials11 and Sackett et al., Evidence-based medicine: how to practice and teach EBM.12

out of the nine criteria for inclusion in this review. The two reviewers assessed the quality of the studies independently. Any differences were resolved by subsequent discussion.

Results Description of Studies The searches located a total of 1,042 articles, of which seventy-nine appeared potentially relevant to the review (i.e., met one or more of the criteria for considering studies in this review) and were retrieved. Many articles were immediately excluded, such as letters, review articles, editorials, and articles merely describing the development of CAL programs in dentistry without formally evaluating their effectiveness. Twenty-seven articles meeting the criteria for consideration in this review were formally assessed for inclusion by two independent raters. The level of initial agreement on inclusion between reviewers was good, indicating that the inclusion criteria could be reliably applied (average Kappa score for agreement on decisions = 0.86). All disagreements were resolved by discussion. Fifteen of the articles were excluded,1327 while twelve articles met the criteria for inclusion2839 in this review. Included articles featured CAL pro-

526

grams in various dental disciplines: five in endodontics, three in orthodontics, one in oral anatomy, one in restorative dentistry, one in geriatric dentistry, and one in prosthodontics. The average score out of nine criteria for the included studies was 7.5. One study met all nine criteria,37 and three studies met eight30,32,34 out of the nine criteria (Table 2).

Methodological Quality of Included Studies All of the included studies compared the effectiveness of a computer-aided learning group (also referred to as a simulation group) with another type of learning strategy such as tutorial teaching, seminar group teaching, lectures, or self-instruction without the use of computers. The study designs that provided the best level of evidence were those in which comparison groups were homogeneously distributed using an academic skill indicator (pre-test scores, cumulative GPA, and GPA in courses related to the CAL program) and subsequently randomly allocated to the comparison groups (Figure 1). This study design was seen in trials by Fouad et al.,30 Mullaney et al.,34 Mulligan and Wood,35 and Puskas et al.37 Studies not following this design that were found to provide an acceptable level of evidence (greater than seven out of the nine criteria on the quality assessment checklist) either adjusted for differences between comparison groups or performed statistical analysis to ensure homogeneity of groups. To ensure that the comparison groups did not differ in learning ability, trials by Bachman et al.,28 Clark et al.,29 Hobson et al.,31 Plasschaert et al.,36 and Sandoval et al.38 performed statistical analysis of pre-test scores between groups following randomization of subjects into groups while another study by Tira.39 reported adjusted post-test scores for pre-test scores. A randomized controlled study by Kay et al.32 used a Solomon three-group design, which allows for comparisons of the effects of differences in variability between groups and is able to identify any discrepancies in skill level between the groups. In another study by Luffingham,33 both groups were exposed to the two types of learning methods (CAL and tutorial), and test scores were then compared between the two groups to detect differences.

Quantitative Results All quantitative results for the studies included in this review are summarized in Table 2. The effec-

Journal of Dental Education ■ Volume 67, Number 5

May 2003

■

Journal of Dental Education

527

continues on next page

Table 2. Summary of included studies and quantitative results. Included studies had to meet at least seven out of nine criteria (listed in Table 1)

Table 2., continued

528

Journal of Dental Education ■ Volume 67, Number 5

Figure 1. Study design to determine the effectiveness of CAL programs versus other teaching methods. Trials with this design provide the best level of evidence for the effect of CAL on academic performance. tiveness of computer-aided learning programs was assessed in all of the included studies by measuring the performance of participants on a post-intervention test or procedure and comparing these scores between the experimental and control groups. Five of the included studies documented statistically significant differences in the outcome measures of multiple choice test scores,30 written test scores,33,34,39 and clinical performance38 favoring the Computer-Aided Learning group over the comparison group with pvalues ranging from 0.001 to 0.05. Six of the included studies revealed no statistically significant difference in outcome measures between the CAL group and the comparison(s) groups.28,29,32,35-37 One of the studies included in this review documented a greater improvement in test scores in the comparison group (seminar groups) over the CAL group (p