Assessing Computer Literacy and Achievement Using Self-Concepts

Assessing Computer Literacy and Achievement Using Self-Concepts William K. LeBold, Robert Montgomery, Dan Budny, Goranka Bjedov and Sherman Ward Purdu...
Author: Jocelin Cooper
0 downloads 2 Views 57KB Size
Assessing Computer Literacy and Achievement Using Self-Concepts William K. LeBold, Robert Montgomery, Dan Budny, Goranka Bjedov and Sherman Ward Purdue University Abstract The Computer Development Self-Appraisal Survey (CDS) was developed to measure computer literacy, knowledge and competency. The CDS scales have high reliability and differential validity. This report summarizes the results of assessing the computer backgrounds and achievements of over a thousand 1996 Purdue first year engineering students who completed the CDS at the beginning and end of their first college computer course. Highly significant gains were observed on all of the CDS computer scales. In-depth analysis of item data document significant gains not only in content related to the courses completed, but also general gains in computer literacy not so related. The report also reflects some significant gender differences, but only minor geographic and ethnic differences.

Background As a result of a decade of studies of computer development [6,7] and student self concepts, the authors and their colleagues are convinced that student self-reports can provide valuable information that can be used in assessing the backgrounds, interests and achievements of students and graduates [1,2,3,4,5]. Purdue’s educational research and information systems group has used self-reports to obtain information regarding courses, grades, honors, activities, skills, interests, demography and a wide range of cognitive and affective information [2]. Using self-reports to measure student backgrounds and achievements in mathematics and chemistry have been a critical part of Purdue’s program to place students in beginning mathematics and chemistry courses and to assess their growth in these beginning courses [3,4]. Although there is some concern often expressed regarding the value of self-reported information, many studies have indicated that self-reports produce valid and reliable information comparable to official records and formal testing [5]. Baird, in his excellent Educational Testing Service Research Monograph has pointed out, “self-reports can be believed, they can be made psychometrically adequate and useful, and often are more predictive of later creative accomplishments than grades and test scores”. Earlier studies supported by departmental [6] and externally funded research studies [7] indicated self reports were especially valuable in assessing cognitive abilities including computer literacy, knowledge and competency. In view of increasing concern regarding assessment as an integral part of future accreditation of engineering

education programs [8] and the critical role of computers in engineering, assessment of computer development seems especially timely and relevant. The Computer Development Self-Appraisal Survey (CDS) was developed in 1986 to measure the computer backgrounds of beginning Purdue Engineering students and their self-concepts. The CDS has 150 items, including 100 items that focus on computers and 50 items that focus on self-concepts. Initially students were re-administered the same 150 items after completing their first computer course. Later the re-administration included the same 100 items included in the pre-testing, but 50 items that focused on the content of the initial computer course: UNIX and computer programming terms. In 1995 the content was changed to reflect many of the newer emerging computer developments (i.e. Internet, workstations, etc.), but the 50 self-concept items were retained. The 50 items on the post test were modified to reflect changes in initial course content, including spreadsheets and Matlab. This report focuses on the results of using the newer versions of the Computer Development Survey and the current computer courses taken by Purdue engineering students. Objectives The primary purpose of this study was to assess the computer backgrounds, computer achievements, and self concepts of beginning engineering students using the Computer Development Self-Appraisal Survey (CDS). The specific objectives are: 1. To assess beginning engineering students’ computer literacy, knowledge and competency. 2. To assess the beginning engineering students’ self concepts. 3. To assess gains in computer literacy, knowledge and competency of students after completing their first computer course. 4. To assess student perceptions of their knowledge regarding the content of their initial computer tools course and their relationship to computer grades earned. 5. To assess differences in student conceptions of their computer literacy and self concepts by gender, ethnicity, geographic origin, mathematical ability, and first semester grade point average. 6. To assess the value of the Computer Development Self-Appraisal Survey in examining the computer

development and self-concepts of engineering students. Method Over 1500 students who were beginning 1996 Purdue engineering students were given the Purdue Interest Questionnaire and the Computer Development SelfAppraisal Survey in the initial class of weekly engineering orientation lectures and courses. The students were instructed to return them the following week; most students returned the survey as scheduled. Usable returns on the pre-test (CDS) were obtained from 1435 of 1573 (91%) beginning engineering students enrolled in the orientation courses. At the end of the first semester, students enrolled in three beginning engineering courses were asked to complete the CDS again. Usable returns on the post test were obtained from 1103 of 1403 (79%) beginning engineering students still enrolled in their first computer course. For students enrolled in the regular Introduction to Computer Tools Engineering course (ENGR 106) and the Honors Engineering Computer Tools course (ENGR 116), the post test CDS form included the same 100 computer items as the pre test but the 50 self concept items were replaced with 50 computer items related to the course content (UNIX, Matlab, etc.). In the Software Tools Tutoring course (ENGR 195S) the same 50 self concept items used in the pre test CDS form were used since most of the content of the Computer Tools courses ( ENGR 106 & 116) were not covered in ENGR 195S. Raw scores were computed for each scale by weighting the responses in hierarchical order: 1 for the lowest response, 2 for the next...and 5 for the highest response. Cronbach’s Alpha was used to measure internal consistency reliability. To facilitate comparisons between scales, all raw scores were converted to T scores (Mean = 50: Standard Deviation = 10) using the beginning 1995 engineering students as the base population. Copies of the pre and post test exams are available through the authors.

Assessing Computer Literacy Examination of the pre test results indicates that incoming students’ assessment of their computer literacy and knowledge is limited. They know about and can use personal computers and some related software, especially word processing. Students are generally aware of and have used the Internet and the World Wide Web communications and have used computers at home, at school, and to play games. But most students are unfamiliar with most computer languages (except BASIC), operating systems (except Windows), computational graphics systems (except Excel and Lotus 1-2-3), and are unaware of most computer terms and editors. The post test

indicates some general growth in computer literacy and knowledge in all areas, especially in operating systems (UNIX), software (e-mail), computational systems (Matlab), most computer terms, editors, and communications (e-mail, World Wide Web). Students still use computers for games and other uses and are more likely to use computers for school and and network access, but their use of computers for programming and language remains relatively low.

Assessing Self Concepts With regard to self-concepts, the pre test indicates most students have relatively high self-perceptions. For example, under Academic Ability Academic Ability: only 3% felt they are below average in their academic ability, 97% felt they are average or above, 90% above average, 68% in the highest 25% and 32% felt they are in the highest 10% of students their age! Similar high perceptions were held by beginning students of their mathematics, and problem-solving abilities. Beginning engineering students are also confident of their ability to work independently. Their drive to achieve, and their intellectual self-confidence and work self-confidence are quite high. The only areas in which the majority of the incoming students did not think they are above average was in their computer programming and artistic abilities, but still average or above when compared with students of the same age!

Assessing Gains in Computer Knowledge Table 1 summarizes the results of the overall computer scale results including the mean T scores at the beginning and end of the semester. Table 1 includes the Mean scores, Standard Deviations, Mean Differences, the Effect Sizes (Ratio of Difference to Standard Deviation) for all students who completed both the pre and post tests. These data reflect highly significant statistical positive changes on all computer scales with effect sizes ranging from .1 to .6. The greatest changes were in Computer Literacy, Large Scale Systems, Editors, and Communications, and the smallest changes in Computer Languages and Personal Computers. Literacy was operationally defined as the number of 80 computer items that the students had at least “Heard About it.” Computer Languages were not covered in these beginning computer classes. Personal computer use remained high, but since our courses are not taught on PC’s, only slight increases were reported. Table 2 provides some further insight into the computer growth by the computer course completed. Students in all courses showed overall growth in Literacy, Knowledge, and Competency, but none reported growth in their understanding of Computer Languages. In the Software Tutoring course (ENGR 195S) the major gains

were in understanding Large Scale Systems, Editors, and Communications; since this course focused on UNIX, editors, e-mail and the World Wide Web these changes are understandable. In the regular Computer Tools course (ENGR 106) highly significant positive changes were observed on all scales, except Languages. For ENGR 106 assignments these highly significant changes are understandable since only languages were not covered in the beginning computer tools courses. In the Computer Honors Tools course (ENGR 116) the greatest gains were

in Large Scale Systems, Computational Systems, and Communications; in view of the emphasis given to UNIX, Matlab and the use of e-mail and the Web for ENGR 116 assignments, these highly significant changes are understandable Relating Grades and Computer Assessments Table 3 relates the grades received in Engineering 106 (Computer Tools) and ENGR 116 (Honors Computer

Table 1-Changes in CDS scores for 1996 Engineering Students who Completed a 1st Semester Computer Course

N=1022

MEAN

SCALE

PRE

STD. POST

DIFF

EFFECT

DEV.

SIZE

OVERALL

50.4

54.4

4.0

9.2

.45

d

COMPETENCY

50.3

53.6

3.3

9.9

.35

d

KNOWLEDGE

50.6

55.0

4.3

9.2

.50

d

LITERACY

50.1

55.5

5.4

9.1

.60

d

LANGUAGES

49.7

50.8

1.1

12.4

.11

b

LARGE SCALE SYSTEMS

50.2

54.9

4.7

10.7

.55

d

PERSONAL COMPUTERS

50.4

52.2

1.9

9.4

.19

d

SOFTWARE

50.8

53.9

3.1

9.5

.32

d

COMPUTER GRAPHICS

50.3

54.4

4.1

.46

d

TERMS

50.1

52.3

2.2

.22

d

EDITORS

49.1

54.7

5.7

.61

d

COMMUNICATIONS

52.1

57.0

4.9

8.9

.56

d

ACTIVITIES

50.2

53.1

2.9

8.9

.30

d

10.6 9.4 10.7

t test b= p

Suggest Documents