Emergency Medical Technician Test Score Analysis 1

Emergency Medical Technician Test Score Analysis1 Rural Emergency Medical Services Outreach Project and University Center for Economic Development 1...
0 downloads 0 Views 376KB Size
Emergency Medical Technician Test Score Analysis1

Rural Emergency Medical Services Outreach Project and University Center for Economic Development

1

Also available as University of Nevada, Reno Center for Economic Development Technical Report Number UCED 2004/05-04

4/1/2004

Table of Contents

Overview......................................................................................................2 Withdrawal Rate for All Students................................................................3 Average Test Score for All Students ...........................................................4 Location .......................................................................................................5 Class Level ..................................................................................................7 Type of Instruction ......................................................................................8 Class Duration ...........................................................................................10 Contract Classes.........................................................................................12 Class Size ...................................................................................................13 Instructor/Coordinator Effects ...................................................................15 Regression Results .....................................................................................16 Conclusion .................................................................................................18

1

4/1/2004

Overview This report summarizes the findings of an analysis of student scores on the standardized State of Nevada, Emergency Medical Service (EMS) Certification Basic and Intermediate level tests. These students had taken emergency medical technician training classes held throughout rural northeastern Nevada from fall of 2001 through winter of 2003. The purpose of this report is to explore whether various attributes of the classes, such as; live instruction or interactive video (IAV) presentation of material; classroom size; or class location; appear to effect student outcomes. Student outcomes are reflected by either withdrawal from the class or average test scores. The Rural EMS Outreach Project at Great Basin College in Elko, Nevada collected data on classroom attributes and test scores. The test scores were analyzed and compiled into a University Center for Economic Development report. In this report, classroom attributes are described, followed by a comparison of average, median, high and low test scores for the differing attributes. Withdrawal rates are also compared for the various classroom attributes. A regression analysis was completed in order to isolate the contribution of the different classroom attributes to test scores. One instructor coordinated all of the IAV classes. The regression model predicts higher average scores for the students in these classes, both in the originating and receiving IAV classrooms. In this model, which isolated the effect of live versus interactive video presentation, the difference between the average scores for the two groups was found to be negligible. When measured by test score, IAV classes are as successful as traditional live classes. Students in a special nine-month extended class had especially high-test scores, and this was also reflected in the regression model. The success of these classes deserves further exploration. John Nolan, Director Nevada Rural Emergency Medical Services Outreach Project Great Basin College Elko, Nevada Elizabeth Fadali and Thomas R. Harris, Ph.D. University Center for Economic Development Department of Applied Economics and Statistics University of Nevada, Reno

2

4/1/2004

Test Score Analysis Class Attributes Withdrawal Rate for All Students A total of 284 students in 28 EMT training classes were included in the test score analysis. These students attended classes held throughout Northeastern Nevada from Fall of 2001 through Winter 2003. Of the 284 students included in the analysis, over 85% (243 individuals) completed the training classes (Figure 1). Table 1 shows how the withdrawal rate (one minus the completion rate) might vary if we assume that the 284 students are a random sample of EMT students and classes in Northern Nevada. Using a 95% confidence interval, the withdrawal rate would be between 10.3% and 18.5% for similar Northern Nevada samples of classes. The test scores used in this analysis are the scores for the 243 individuals who completed the program.2 Figure 1. Percent of Students Completing Program

Completion Withdrew 14.4%

Completed 85.6%

Table 1 Withdrawal Rate for Total Sample of EMT Students Total

% Withdrawals 14.4%

95% Confidence Interval 10.3% to 18.5%

Number of Students 284

2

Information on two scores was missing although it is known that the individuals successfully completed the program. Thus the total group used to compute averages is actually only 241 students.

3

4/1/2004

Average Test Score for All Students Students in the EMT training classes take a standardized statewide EMS Certification test. The certification test is administered by the State of Nevada EMS office with grading and testing taking place at the local branch of the State of Nevada EMS office. Students must already have obtained their CPR certification before they enter the training class and are otherwise screened before they are accepted into the training classes. Additionally, students typically withdraw if they realize they have not put in enough time and effort to pass the test. Thus, only one score was recorded that was lower than the minimum of 70 that is needed to pass (a 66, the minimum score). The lowest grade amongst the rest of the group is the minimum needed to pass, a 70 (12 students or 5% scored a 70 on the test). The highest score recorded was a 98. The average score for the entire group of 241 students (2 of the 243 students were missing test score data) was 82.5. The median score was similar at 83. A 95% confidence interval for the overall average test score for similar samples would be 81.6 to 83.4. These results can are shown in Table 2. The histogram in Figure 2 shows that about 48% of the scores are bunched at midrange from 77 to 87. Figure 2. Histogram of Student Grades

Student Grade Distribution

Percent

30%

20%

10%

0% 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96

Student Grade on EMT Test

Table 2 Average, Median and Range of Test Scores for Total 241 Students

Total

Average Score 82.5

Median Score 83

Low Score High Score 95% Confidence Interval 66 98 81.6 to 83.4

Number of Students 241

4

4/1/2004

Location As seen in Figure 2 the EMT classes took place throughout Northeastern Nevada, although the majority of students attended class at Great Basin College in Elko (54.6%). The remaining 45.4% of the students attended classes at Winnemucca, Wendover, Owyhee, Eureka, Ely, Austin and Battle Mountain. Figure 1. Location of EMT Students

Location Winnemucca 7.0% Wendover 7.4% Owyhee

Austin 2.8% Battle Mountain 2.8%

5.6% Eureka 8.5% Ely 11.3%

Elko 54.6%

5

4/1/2004 The average test score for students whose classes took place at Great Basin College was 82.3, not substantially different than the average for students whose classes took place elsewhere in Northeastern Nevada (82.8). The withdrawal rate, however, was substantially higher for the off-site locations (23.3%) as compared to Great Basin College classes (7.1%). Off-site classes had the highest rate of withdrawal of any class attribute. Table 3 compares median, high and low scores as well as 95% confidence intervals for the average score and numbers of students in each category. Table 4 compares withdrawal rates in a similar manner. Figure 2 Test Score Average and Withdrawal Rate by Location

Location Comparison 25.0% 23.3%

96

20.0%

86

82.8

82.3

15.0% Average Score 10.0%

76

% Withdrawals

7.1% 5.0%

66

0.0% Elko

Offsite

Table 3 Comparison of Test Scores by Location

Elko Offsite

Average Score 82.3 82.8

Median Score 83 82

Low Score High Score 95% Confidence Interval 66 97 81.1 to 83.5 70 98 81.4 to 84.2

Number of Students 143 98

Table 4 Comparison of Withdrawal Rate by Location Elko Offsite

% Withdrawals 7.1% 23.3%

95% Confidence Interval 3.0% to 11.2% 15.9% to 30.6%

Number of Students 155 129

6

4/1/2004

Class Level Students included in the analysis were either in Basic or Intermediate level EMT training classes. Most students were in Basic level classes (70.1% or 199 students). From Figure 3, withdrawal rates were similar for both types of classes and were close to the overall average of 14.4%. The average test score was significantly higher for the intermediate level students (86.5) when compared to the average student test score in the basic classes (80.8, Figure 3 and Table 5). Intermediate and basic level students did not take the same test, however. The type of students in intermediate classes may be quite different as well since those going on to this higher level may have more real-life experience with EMS, a higher degree of interest in the subject and higher levels of motivation. Figure 3. Test Scores and Withdrawal Rate by EMT Class Level

Class Level Comparison 25.0% 96 20.0% 86.5 86

14.6%

14.0%

15.0%

80.7 10.0%

Average Score % Withdrawals

76 5.0%

0.0%

66 Basic Level

Inter. Level

Table 5 Comparison of Test Scores by Class Level

Basic Level Inter. Level

Average Score 80.7 86.5

Median Score 81 88.5

Low Score High Score 66 96 70 98

95% Confidence Interval 79.7 to 81.7 84.9 to 88.1

Number of Students 167 74

Table 6 Comparison of Withdrawal Rates by Class Level Basic Level Inter. Level

% Withdrawals 14.6% 14.0%

95% Confidence Interval 9.7% to 19.6% 6.5% to 21.4%

Number of Students 198 86

7

4/1/2004

Type of Instruction About 14% of the group (40 students) were taught long distance via an interactive video system. An instructor or instructors taught courses in person for about 86% of the students with 35% of total students in classes in which the interactive video instruction originated. The remaining 51% of the students were in traditional live classes without any interactive video component. See Figure 4. In one case, a live instructor repeated some of the class instruction at a remote IAV location. These students were included in the IAV instruction group. Figure 4. Students by Type of Instruction

IAV, Live with or without IAV component Live with IAV 35.2% IAV 14.1%

Live without IAV 50.7%

Perhaps surprisingly, the average test score for the interactive video students was higher than the average for students benefiting from a live presentation, either with or without interactive video connection (see Figure 5). In the case of traditional live classes versus interactive video classes the difference in scores was even significant (see Table 7 below for 95% confidence intervals). In addition, withdrawal rates in the traditional classes were rather high at 16.7%, although in this case they were not significantly different from the withdrawal rate in the interactive video sections (15%, see Figure 5 and Table 8). It may be that other factors, such as small class size, instructor/coordinator experience or teaching techniques, are influential in determining withdrawal rates and scores. Average class size for IAV classes was five students, for live classes without IAV average class size was about 12 students and for live classes with IAV, about 20 students.

8

4/1/2004

Figure 5 Test Scores and Withdrawal Rate by Type of Instruction

Comparison by Type of Instruction 25.0% 96 20.0% 16.7%

85.4

86

84.7

15.0%

15.0% 11.0%

Average Score % Withdrawals

80.1 10.0%

76 5.0%

0.0%

66 IAV

Live with IAV

Live w/o IAV

Table 7 Comparison of Test Scores by Type of Instruction

IAV Live with IAV Live w/o IAV

Average Score 85.4 84.7 80.1

Median Score 87 86 80

Low Score High Score 72 98 70 97 66 93

95% Confidence Interval 82.8 to 87.9 83.1 to 86.2 78.9 to 81.2

Number of Students 33 89 119

Table 8 Comparison of Withdrawal Rates by Type of Instruction IAV Live with IAV Live w/o IAV

% Withdrawals 15.0% 11.0% 16.7%

95% Confidence Interval 3.8% to 26.2% 4.8% to 17.2% 10.6% to 22.8%

Number of Students 40 100 144

9

4/1/2004

Class Duration All classes had to meet minimum requirements for hours of instruction. The minimum instruction hours are 110 hours for Basic training and 63 additional hours for Intermediate training. Each level requires 10 additional hours of clinical practice. However, instruction in some cases was compressed into as little as two weeks or extended as long as a school year. Most students (55.6%) took classes of approximately a semester duration, here defined to be anywhere from two and one half months to four months long. Nearly 40% of students took classes that were condensed (less than two months duration). About 5% (14 students) enrolled in a special high school course that ran over the course of the school year and included many hours beyond minimum requirements. Figure 6 Percentage of Students by Class Duration

Duration of Class Instruction Extended 4.9% Condensed 39.4%

Semester 55.6%

10

4/1/2004 Withdrawal rate was highest for the condensed classes and lowest for the extended class. This seems reasonable since the condensed classes require a large number of hours in a short time period and thus are very demanding in terms of the student’s daily schedule. Wide confidence intervals produce no significant differences between withdrawal rates, however (Table 10). Average test scores were significantly higher for the extended class (89.8) than either the condensed or semester class averages (83.9 and 80.8 respectively, see Figure 7). There is a significant difference despite the small group in the extended classes (14 students) because the grades were so uniformly high. This is not surprising, given that the extended group had the benefit of many additional hours of instruction. Condensed classes have higher average scores than semester classes as well. This could be because the shorter time period allows for greater focus and retention of material. Over half of condensed class students were intermediate level. Figure 7. Test Scores and Withdrawal Rate by Class Duration

Comparison of Class Duration 25.0% 96 89.8 86

16.1% 83.9

20.0% 15.0%

13.9% 80.8

Average Score % Withdrawals

10.0% 76

7.1% 5.0%

66

0.0% Condensed

Semester

Extended

Table 9 Comparison of Test Scores by Class Duration

Condensed Semester Extended

Average Score 83.9 80.8 89.8

Median Score 84 81 90

Low Score 70 66 85

High Score 95% Confidence Interval 98 82.4 to 85.5 96 79.7 to 81.9 93 88.5 to 91.0

Number of Students 93 135 13

Table 10 Comparison of Withdrawal Rates by Class Duration Condensed Semester Extended

% Withdrawals 16.1% 13.9% 7.1%

95% Confidence Interval 9.2% to 22.9% 8.5% to 19.3% 0.0% to 21.1%

Number of Students 112 158 14

11

4/1/2004

Contract Classes Some of the classes were taught as a result of a contract with a private firm on behalf of their employees, who were then usually paid while attending the course. Forty-one students of 14.4% were in this type of class. The lowest withdrawal rate reported for any class attribute was for classes contracted by private companies. These classes had no withdrawals. This may be because the class is considered part of a work requirement for the students in these classes so they are less voluntary in nature. Test scores in the contract classes were the lowest, with an average of 79, for any class attribute and significantly lower than the average for non-contract classes (83.2). Figure 8. Contract Classes Test Scores and Withdrawal Rate Comparison

Comparison of Contract and Non-contract Classes 25.0% 96 20.0% 16.9% 86

83.2

15.0%

Average Score % Withdrawals

79.0

10.0%

76 5.0% 66

0.0%

0.0% Contract

Non-contract

Table 11 Comparison of Test Scores for Contract and Non-contract Classes

Contract Non-contract

Average Score 79.0 83.2

Median Score 79.5 83

Low Score 70 66

High Score 91 98

95% Confidence Interval 77.2 to 80.8 82.2 to 84.2

Number of Students 40 201

Table 12 Comparison of Withdrawal Rates for Contract and Non-contract Classes Contract Non-contract

% Withdrawals 0.0% 16.9%

95% Confidence Interval 0.0% to 0.0% 12.2% to 21.6%

Number of Students 41 243

12

4/1/2004

Class Size Class size varied from one individual in a location receiving interactive video instruction to a class of 23 students. Average class size was about 14 students per class. Median class size was a little higher at 15 students per classroom. Class size as defined in this report includes only individuals in one location. For example, if 20 students received live instruction at Great Basin College, but the instruction was broadcast on interactive video systems to two other locations, one which has five students and another with seven students, these were recorded as three separate classes of size twenty, five and seven. About 9% of total students were in very small classes with one to five students (25 students). About 25% were in classes of six to ten students, 24 % in classes of eleven to fifteen students, while 42% were in the largest classes of over sixteen students. Figure 9. Percentage of Students by Class Size

Class Size More than Sixteen 42.3%

One to Five 8.8%

Six to Ten 24.6%

Eleven to Fifteen 24.3%

13

4/1/2004 Withdrawal rates were lowest for the students in the smallest class size, second lowest for the largest class size (16 students and up) and highest for the six to ten category (Figure 10). Only the one to five class size and the six to ten class size had a significant difference between withdrawal rates (Table 14). Average test scores followed a similar pattern with the highest average observed for the students in the smallest class size, and next highest average observed for the students in the largest class size. For test scores, a significant difference was found between the average score for the students in the smallest group and those in the classes of 11 to 15 students (Table 13). Figure 10 Test Scores and Withdrawal Rate by Class Size

Class Size Comparison 25.0% 96

21.4%

20.0% 17.4%

86.3 86

82.0

81.1

82.7

15.0%

10.8% 10.0%

Average Score % Withdrawals

76 5.0%

4.0% 66

0.0% 1 to 5

6 to 10

11 to 15

16 and up

Table 13 Comparison of Test Scores by Class Size

Size 1-5 Size 6-10 Size 11-15 Size 16 and up

Average Score 86.3 82.0 81.1 82.7

Median Score 88 82 81 83

Low Score 72 70 66 70

High Score 95% Confidence Interval 98 83.2 to 89.5 96 80.1 to 83.8 96 79.3 to 82.9 97 81.3 to 84.0

Number of Students 23 55 56 107

Table 14 Comparison of Withdrawal Rates by Class Size Size 1-5 Size 6-10 Size 11-15 Size 16 and up

% Withdrawals 4.0% 21.4% 17.4% 10.8%

95% Confidence Interval 0% to 11.8% 11.8% to 31.1% 8.4% to 26.4% 5.3% to 16.4%

Number of Students 25 70 69 120

14

4/1/2004

Teacher Comparison Teacher attributes, such as years of experience, are not available from the dataset. In fact, classes may have many contributors from the healthcare community and, as such, cannot be said to be taught by one person only. However, it is known that one teacher coordinated the instruction for all IAV and live classes with IAV components, or about 49% of the total group of students. It is therefore difficult to isolate the effects of the IAV program and the effects of the leadership of a particular teacher. Figure 11 compares withdrawal rates and average scores for this teacher/coordinator (IAV plus live classes with an IAV component) and all other teachers/coordinators (all traditional live classes). The average score for the students with the IAV instructor (84.9) are significantly higher than the average from the students without this instructor (80.1, Figure 11 and Table 15). Withdrawal rates are lower for the IAV instructor’s classes as well (12.1% versus 16.7%, Figure 11), but in this case the difference is not significant (Table 16). Figure 11 Test Scores and Withdrawal Rate by IAV Teacher

Teacher Comparison 25.0% 96 20.0% 16.7%

84.9

86 80.1

15.0%

Average Score % Withdrawals

12.1% 10.0%

76 5.0% 66

0.0% Other teacher

IAV Teacher

Table 15 Comparison of Test Scores by IAV Teacher

Other teacher IAV Teacher

Average Score 80.1 84.9

Median Score 80 86.5

Low Score High Score 66 93 70 98

95% Confidence Interval 78.9 to 81.2 83.5 to 86.2

Number of Students 119 122

Table 16 Comparison of Withdrawal Rates by IAV Teacher Other teacher IAV Teacher

% Withdrawals 16.7% 12.1%

95% Confidence Interval 10.6% to 22.8% 6.7% to 17.5%

Number of Students 144 140

15

4/1/2004

Regression Analysis A regression analysis was carried out on the test scores variable. Regression analysis is important because it allows us to isolate the effect of one variable from several other variables. For example, it may be difficult to tell whether scores are different because of class size alone or because they tend to be correlated with other class attributes, say, being taught with interactive video or being “offsite” (not on the Elko campus). Results of the regression model are reported in Table 17. Only those students who had received a grade were included in the analysis, so the number of students included was 241. Results are listed in Table 17. Coefficients in this model give the average change in test score for a unit change in the classroom attribute. For example, our model predicts that if the number of students in a class increases by one, average test scores will increase by .08 points, if all other classroom attributes are held constant. Similarly, coefficients on indicator variables give the increase or decrease in average test score for students in classes with that attribute compared to the average for students in classes without that attribute. For example, if students are in an extended class, the model predicts that the average for those students will be 11.86 points higher than the base case, being in an intermediate class, when all other attributes are held constant. The probability, p, given in the significance column indicates how likely it is that the coefficient is not statistically different than zero (that is, has no influence on test scores). In this model only the indicator variables on IAV teacher, extended and condensed classes are significantly different than zero at the 95% confidence level. All these attributes have a positive influence on test scores. Students who have the IAV instructor coordinate their class will have an average score 5.37 points higher than students who do not have this instructor coordinate their class, all else being equal. Students who are in either the extended or condensed classes have higher average scores than do those in the semester length classes. The rest of the class attributes do not contribute conclusively to average test scores. Table 17 Results of Regressions on Test Scores (Constant) Offsite Taught in person IAV teacher Contract with private company Number of students in classroom Intermediate level Extended Condensed Adjusted r-squared

Coefficient (B) Standard Error 74.81 3.63 1.94 1.88 0.55 3.03 5.37 2.22 -2.66 2.39 0.08 0.15 1.12 1.39 11.86 2.07 5.12 1.39 28.9%

Significance (p) 0.00 0.30 0.86 0.02 0.27 0.59 0.42 0.00 0.00

The results of the regression in some cases give a different picture than the attribute-byattribute comparisons. For example, in comparing the average score for students in classes contracted by private companies and those not in such classes, there is a significant difference in scores. Although the regression model predicts a lower average score for contract classes, the difference is no longer significant when other factors are taken into account. The difference in average scores for students in intermediate and basic level classes, though still positive, also loses significance when other factors are taken into account. 16

4/1/2004 The r-squared value is 28.9%, which means that 28.9% of the variation in test scores could be explained by this regression model. The majority of the variation in test scores comes from unexplained factors outside of the model. Some of these unexplained factors may be quite important explanatory factors such as years of experience of instructors/coordinators and teaching techniques used as well as the gender, age, years of education or other socio-economic characteristics of the students. Teacher characteristics, differing classroom techniques as well as the socioeconomic characteristics of students, all of which might have important impacts on test scores, were not included in the model because data on these aspects was not available in this dataset.

17

4/1/2004

Conclusion A comparison of student scores on the standardized EMS Certification tests shows significant differences in averages for students in different classroom situations. In some cases, significantly different withdrawal rates are found as well. The following class attributes were associated with significantly higher average test scores: •

Taking an intermediate level class.



Interactive video presentation or live class with originating interactive video presentation. These attributes were synonymous with one coordinator’s leadership and methods.



Both condensed classes and the extended nine-month high school classes had higher average scores than semester length classes. Students in the extended classes had especially high average scores.



Non-contract classes.



A small group of students in very small class sizes (mostly interactive video students) had higher averages than students in class sizes from eleven to fifteen.

Withdrawal rates were significantly higher for class sites off the Great Basin Campus. The classes that had been contracted by private companies for their own employees had no withdrawals, the lowest withdrawal rate for any class attribute. A regression model predicts higher average scores for those students whose classes are coordinated by the instructor who has been in charge of all interactive video presentations in the area. In this sample, students taught via interactive video did not have lower average scores and live presentation had negligible benefits over interactive video presentations. The model predicted average scores for students in the extended class to be 12 points higher than students in semester classes. Condensed classes were predicted to have an average score five points higher than semester length classes. Other factors were found to have a negligible effect. Teacher characteristics, differing classroom techniques as well as the socioeconomic characteristics of students, all of which might have important impacts on test scores, were not included in the model because data on these aspects was not available in this dataset.

18

Suggest Documents