Session 13c3 Using Case Studies to Evaluate Learning Technologies Julie Baher and Joyce Ma Northwestern University The Institute for the Learning Sciences 1890 Maple Ave. Evanston, IL 60201 Abstract- Qualitative methods are key tools for analyzing complex interactions and environments. Yet little qualitative work has been done, especially in higher education, to develop a deep understanding of technology use in the classroom. In this paper we will discuss how we have used case studies to evaluate the role of learning technologies in engineering education. Using qualitative research methods, such as classroom observations, interviews and field research, we have looked at diverse topics ranging from technology adoption to cognitive change. In this paper, we present examples of our work to illustrate how these methods are used in diverse settings to understand complex education phenomena.

research, we are developing an understanding of what the challenges are in re-designing curriculum to incorporate a new technology. Case study research on FAVL has been used to elicit how high school students’ conceptions of feedback systems evolve in the course of students’ work with the FAVL software and curriculum. [7] Using interviews and field observations of students working with FAVL, we are developing a model of cognitive change that will inform the design of FAVL as well as other computer-based learning environments. In both studies, we have used qualitative research methods to develop an understanding of complex phenomena or processes that could not be captured through quantitative methods alone.

Introduction What are case studies? Much of the current research on engineering education uses quantitative methodologies. In particular, studies use instruments such as course evaluations, tests and exams and grades to measure, for example, the results of using new software programs to teach students. While these studies offer valuable insight into the effectiveness of an educational technology, the methodology used can only address certain aspects of engineering education. Quantitative assessment cannot provide the depth and richness of detail that are necessary to address complementary questions such as how learning technologies are implemented in the classroom or what the professor’s role is in shaping how these technologies are used by students [1]. Nor can they reveal in detail how students' conceptual understanding evolves during their work with a new educational technology. Several researchers have highlighted this need for a broader based approach to educational studies [2-4]. We propose that qualitative methods, especially case studies, are critical in understanding issues surrounding technology use and effectiveness in the complex environment of the classroom. In this paper we will discuss how we use case studies in our research to develop Articulate Virtual Laboratories, a National Science Foundation funded project. Two products, CyclePad and the Feedback Articulate Virtual Lab (FAVL) have been developed for use in engineering programs under this project [5]. Case study research on CyclePad has been used to examine how professors at several different types of engineering schools (from military to state to private) have implemented the program in their classrooms [6]. From this

The idea of “case studies” encompasses a broad range of writings from reports that chronicle and describe events or test hypotheses to those which are meant to teach [8]. Common to all these types of case studies is that they offer a rich description which the reader can use to draw comparisons to other similar situations. Lincoln and Guba write: … they permit the reader to build on his or her own tacit knowledge in ways that foster empathy and assess intentionality, because they enable the reader to achieve personal understandings … and because they enable detailed probing of an instance in question rather than mere surface description of a multitude of cases.[8] (p358) In this paper, we focus on case studies that are used as a means of conducting and communicating educational research (rather than the type of case studies used as tools for teaching students). External validity for case studies is rooted in the belief that the reader can find opportunities to see the similarity between his/her own experience with the case described and thus make use of the material offered in the case study. In this way a particular case study becomes generalizable to the reader’s own cache of cases. Thus it is necessary to create detailed case descriptions so that the reader can determine what would transfer to his or her own situation. Lincoln and Guba explain that:

0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13c3-12

Session 13c3 the case study provides the “thick description” so necessary for judgments of transferability. … judgements of transferability depend upon a sufficient knowledge base for both sending and receiving context. It is the responsibility of the inquirer to provide a sufficient base to permit a person contemplating application in another receiving setting to make the needed comparisons of similarity. [8](p359) Using Case Studies Below we present two brief excerpts from our work to illustrate how we have used interviews, classroom observations, student work and other artifacts to construct cases of engineering teaching and learning.

Example 1: Studying Teaching Practice In our first example, case studies are used to examine factors in teacher’s cognition and context that may influence how technology is enacted with curricula. In particular, we followed three professors closely as they sought to incorporate CyclePad into their classroom teaching. We chose to use case studies because the complexities and subtleties of the relationships between contextual and cognitive factors that influence adoption proved difficult to elicit with more traditional quantitative methods. In the following, we will briefly describe one of the three cases studied. The Classroom and School Context Professor NJ teaches at a state school with 10,000 full and part-time students and over 400 full-time faculty members. He held a joint appointment with the Mechanical Engineering Technology and Applied Science departments. Professor NJ was aware of the tension between how he was taught thermodynamics, his prior experience at research universities and his current position as a professor in an engineering technology program. The difference in these two fields: engineering science and engineering technology, has implications for how Professor NJ felt engineering technology should be taught. He explained: So almost all of us [the professors] come out of engineering programs. So when you teach a course in thermo you tend to do it the way you were taught. … I don't think engineering technology should just be engineering minus the math. That's to me not worth having. It should be engineering minus the math plus something. And the something is supposed to be practice oriented, hands-on

For this research, we followed Professor NJ’s teaching of Applied Thermal Sciences in 1997 and 1998. This course is an analog to the full year thermodynamics course sequence taught in engineering sciences programs. The class meets in both a regular classroom and, on lab days, in a computer lab. Although there were enough computers for each student to work on one, he had the students work in pairs so that they could “learn from each other.” In the classes that we observed the students were predominantly white and male and the class-size was small - between 10 and 12 students. The average age of the students is 27 and many have some experience in the work force. Many see the engineering technology degree as a pathway to a higher paying, more upwardly mobile career. Qualitative Instruments A detailed account of Professor NJ’s experiences with new technologies was developed from in-depth personal interviews, classroom observations and e-mail correspondence. Additionally, we collected physical artifacts related to the course such as course syllabi, textbooks, course specific web pages, student surveys and university publications. Interviews and videotapes were transcribed and coded to look for themes and patterns. Other items such as emails, web pages and his journal articles were analyzed using the same coding scheme developed from the interviews. Findings The case study of Professor NJ revealed an interesting picture of how a new technology is incorporated into classroom teaching. This section provides some highlights from our observations. Representations of thermodynamic concepts By using the CyclePad software, Professor NJ showed his students another way of representing thermodynamic cycles. While the book used certain forms of diagrams and formulas to represent cycles, CyclePad used another form of representation (see Figure 1. A Diesel cycle in CyclePad). Professor NJ had his students create P-v (pressure-volume) diagrams for cycles and then try and build them in CyclePad. The students struggled with the task, yet Professor NJ felt that it was good for the students to try and see the connection between the graphs and the computer so that they understand that the “graphs have meaning.” He told me that he would use the class period to explain how to interpret graphs and CyclePad models and that if the students had to struggle with it themselves they would get more out of the explanation than otherwise.

0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13c3-13

Session 13c3 I think it [the formulas] will mean more to them after they experiment with CyclePad and they say "what affects efficiency?" and then next week, we're going to derive some of those formulas for efficiency and it will show that, "Hey here's this formula that shows that, yes, it is the compression ratio and you found that when you did your experiments.” And I think that [it] will be more meaningful to them, seeing that second, rather than first.

Figure 1. A Diesel cycle in CyclePad Changing the introduction of topics and terminology As CyclePad brought new forms of representation with it, it also encompassed specific vocabulary and concepts. This meant that Professor NJ had to teach this terminology to his students earlier in the term than he would have otherwise. He explained this change: … the first time I tried using CyclePad [1997], the semester had already started and I said “this is great stuff!” It was all sort of done on the fly. So, this year teaching this course I had in mind to use CyclePad, so I introduced some topics quicker and some terminology quicker. So that when they started using CyclePad, they’ve already been exposed to some of that. In particular, when modeling cycles in CyclePad, students needed to know terminology that describes specific processes. While students were familiar with certain terms such as “isothermal” (constant temperature), other terms such as “adiabatic” (constant heat) or “isobaric” (constant pressure) were not familiar to them. We noticed that the students in the 1998 Applied Thermal Systems course were more proficient at using the terms in conversation than the students in the previous year’s course. Knowledge of Student Needs Professor NJ found CyclePad useful because students could experiment with values without having to know or understand the mathematics. This was especially important in this engineering technology program since students had not yet been exposed to calculus and often struggled with algebra. He hoped that later the students would link the intuitions that they gained from using CyclePad with the mathematics. He explained:

This is not to imply that the students did no calculations. CyclePad could do the simpler calculations, but Professor NJ did not instruct the students on where to look in the program to find those answers (although some students found it on their own). Professor NJ had the students do simple calculations by hand (such as efficiencies, net work and net heat) and use CyclePad for complex mathematics (such as those involving exponents). Quantitative studies of technology use tend to focus on whether a software program was implemented or not rather than focusing on the details of how it was used. In this study, taking a qualitative approach helped us see how Professor NJ adapted his teaching to work CyclePad into the curriculum. Often studies focus on student learning without giving voice to the professor who shaped the learning environment. This type of case study can help give professors an understanding of what it might be like to teach with the software.

Example 2: Studying Student Learning As part of our research on teaching feedback concepts to high school students, we looked closely at two high school students’ evolving models of feedback control systems as they worked on a series of design and analysis tasks in the Feedback Articulate Virtual Laboratory (FAVL). There has to date been few studies that looked in particular at high school students’ conception of feedback. Case studies were therefore useful in the formative phase of our study to identify mental models, which students may have about feedback phenomena and the process of meaning construction that students experienced in working with FAVL. Qualitative methodology provided us with the tools with which to undertake such an exploratory investigation by allowing us to obtain rich data that is essential for a detailed understanding of the thought processes of the students. In addition, we found these case studies to be invaluable in our efforts to design more effective learning environments for teaching feedback concepts. With each case we analyzed, we tried to identify the main problems students were having with the software and used these case results to inform our redesign of the FAVL software and curriculum. Our use of case studies, therefore, is in line with the design experiment

0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13c3-14

Session 13c3 methodology Ann Brown proposed for educational research in which design, theoretical progress and practice are intertwined [9]. The Classroom Context This study was conducted at a local high school in an introductory engineering class. This class is part of the Applied Arts Department and is offered, in most cases, as an elective class that can be taken at any grade level. In March 1998, during this study, most of the students enrolled in the class were freshmen. Class size averaged 20 students. There were four sessions held daily, each averaging 42 minutes. Mr. S., a teacher with over 15 years of experience, taught several sections of the class that academic year. He ran this class, called Engineering SmartLab, as a projectbased classroom where teams (usually pairs) of students worked together on different modules rotating from one module to the next every seven days. Some of these modules involved limited computer work. All modules began with a pre-test and ended with the students’ taking a post-test. These tests formed the basis for evaluation for this class. FAVL was incorporated into sessions of the SmartLab class as one of the many modules students can work on. This type of classroom arrangement provided the advantage of allowing FAVL to be easily integrated into the routine of the classroom with minimal disruption that may affect student work. FAVL, however, differed markedly from the other modules because it is entirely computer based. Also, a researcher sat next to the students while they worked on the module. Students could ask questions whenever they encountered difficulty, a ‘feature’that did not come with the other modules. Finally, student work on FAVL was not graded. Instead, all students who agreed to work with FAVL were guaranteed an A for the project. The Students We worked with four students in March 1998. The pair that we will focus on for the remainder of this paper was chosen by Mr. S. who had asked them to participate in an experimental project for Northwestern University. In a side conversation with Mr. S., he mentioned that he chose students who were honors students and whom he thought would be able to handle the work in FAVL. Susan1, one of the students in this group, was a freshman. At the time, she was enrolled in seven classes, four of which were honors classes (Biology, Algebra, History, and English). On a questionnaire she filled out before working on FAVL, she wrote that she believed that in this SmartLab class, “we learn about computers and 1

Pseudonyms are used for the participants in this research study.

programming.” Susan also wrote that she had some experience with computers but did not elaborate on the type of experience. Carl, the other student in this group, was also a freshman. He was taking five honors classes (Biology, Algebra, History, English, and Latin) and the SmartLab course. On his questionnaire, Carl wrote that he believed that he’d learn about “many technologies used in the industrial world.” Carl played with computers in his time away from school. He wrote, “I even put together my own Cyrix based computer.” Qualitative Instruments To piece together a picture of student understanding, we used several means of collecting data: • Pre-FAVL interview. We conducted a one-on-one interview with each student before they began working with FAVL. Each interview took 15 minutes and was intended to explore students’ understanding of feedback systems prior to the intervention. Students were encouraged to talk about and draw pictures of their understanding. The interviews were audio taped and then transcribed. • Audio-taped and video-taped interactions during their design projects. We worked extensively with the students during all the design projects they did in FAVL. We answered their questions, helped them solve problems, and listened to their concerns. Interactions between the students and the researcher were all captured on audio and videotape. In addition, we encouraged Susan and Carl to think aloud during their design project and to discuss their design strategy with each other. Taped interactions between the students were transcribed for analysis. Although Susan and Carl worked together with FAVL, we tried to analyze their individual patterns of understanding which seem to differ at different points in their work. • Post-FAVL interview. We concluded the unit with a 15-minute one-on-one interview with each student. This interview was designed to elicit students’ descriptions of feedback after their experiences in FAVL. Findings from the Case Studies These pieces of data were used to put together a story of how Susan's and how Carl's understanding changed in the course of working with FAVL. Susan’s Model Susan’s model in many ways lacked internal details. For example, before her work with FAVL, when Susan was

0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13c3-15

Session 13c3 asked to describe how a thermostat regulates the temperature in a room, she answered: it’s basically like a fan works… a ceiling fan probably … to make it cooler, it takes the air in and just fans it off the same way and heating probably uses … a heater This 'black box' model persisted. In her post-interview, when asked how the body regulates it's own temperature, Susan answered: it’s because of outside temperature and inside temperature, it kind of balances it. And I think … 98.7 ° would be like a zero point, like an origin of the whole thing. So it probably has to keep it there to … I think what it keeps you like all… this organ and like all the heat and energy leak from the organs and energy taken in and the balance of those two In addition, we noticed that Susan often referred to the concrete, mentioning physical objects in her explanations: Susan: [explaining how human thermoregulation works] the body feels the outside temperature and the brain tells the body to … produce more energy and [in] FAVL… the bimetallic strip like feels the heat and then makes the connection… I mean… feels the cold and makes the connection with the electricity part. It’s the same thing. Susan's experience with FAVL can be characterized by the learning of a particular feedback system with little abstraction to a more general model. This is even though FAVL requires that students work with 'abstract' components such as sensors and actuators without any reference to a physical instantiation. Carl’s Model Carl, on the other hand, from the onset was able to describe systems with the functional components that characterize an expert’s model of feedback: sensor, comparator, actuator, and setpoint: Carl: [describing room thermostatic control during the pre-interview] probably you would have the thermostat or some sort of particle monitoring mechanism that would sense [sensor] the temperature, … that would sense that the amount of movement of the particles in the air and when they reach a certain speed [comparator], you know, a degree of movement back and forth, the thermostat would shut off or if they were moving too slow, it would produce more heat [actuator] in the air and wait for it to

reach … I guess there’s some sort of mathematical number programmed into the computer so when... the particles move at a speed, it shuts off [set point] We also noticed that Carl’s description became increasingly characterized by the use of the more abstract terms. Carl: the sensor senses the temperature in the room … the SPU which checks to see if it in fact is the certain temperature which turns the on and off switch either on or off to activate the furnace which is the actuator in this case Later, during the post-interview, Carl described the human thermoregulation system: you’ve got, let’s say, internal sensor that compares it to a set point unit. Let’s say that that’s 98.6 is the value here. So, it constantly compares … if that matches that or if that’s over that. … the actuator heats your body up. Carl's experience with FAVL can be characterized by increased proficiency with technical terms and learning to further abstract the functional model for feedback systems so that he can apply them to other examples of such systems. These results show how a case study approach can describe student’s cognitive change. Had we measured their knowledge through a multiple choice test or another quantitative measure, we would have less understanding of the degree to which the students understand the concepts (at what level of abstraction) or specifically how their knowledge changed over the course of instruction. Implications on Designing Learning Environments A comparison between Susan and Carl's cases indicate that students can come to FAVL with very different prior knowledge and mental models. As one of the results of these case studies, we have redesigned the FAVL software and curriculum in order to provide students with more opportunities to learn a concrete and easy to understand example before working with the FAVL software.

Discussion & Conclusion This paper presents brief excerpts from the cases we have developed for our research. What we hope they illustrate, is how qualitative inquiry can help us develop richer understandings of complex human phenomena. There are several excellent sources [1] [8] [10] [11] [12] [13] which offer information on how to conduct qualitative research from study design to data collection, analysis and case reporting.

0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13c3-16

Session 13c3 [9] Our motivation for this paper was to start a conversation about expanding the methodologies used in engineering education research so that there would be a greater variety to inform us all. We feel that professors who are interested in conducting educational research in their classroom could use case study methodology to report on formative studies of educational reforms, descriptions of student understanding or of their own personal experiences teaching a new curriculum. These types of studies are also useful for situations in which the quantitative methods would not be possible (such as when the sample size is small or there is no control group). Furthermore, case studies can help us capture important information about how to implement educational change – the type of information that often remains tacit. We feel that case studies could be an important addition to the many quantitative studies that are currently done.

[10]

[11]

[12]

[13]

Acknowledgements

A. L. Brown, "Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom Settings," in The Journal of the Learning Sciences, Vol. 2, No. 2, 1992, pp. 141-178. M. B. Miles and A. M. Huberman, Qualitative Data Analysis. Thousand Oaks, CA: Sage Publications, 1994. R. L. Crowson, “Qualitative research methods in higher education,” in Qualitative Research in Higher Education: Experiencing Alternative Perspectives and Approaches, C. Conrad, A. Neumann, J. G. Haworth, and P. Scott, Eds. Needham Heights, MA: Ginn Press, 1993. D. F. Lancy, Qualitative Research in Education: An introduction to the major traditions. New York: Longman, 1993. R. K. Yin, Case study research: design and methods. Newbury Park, California: SAGE Publications, Inc., 1989.

This research is supported the Applications of Advanced Technology Program, National Science Foundation. Thanks to all the professors, teachers and students who participated in these studies.

References [1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

S. B. Merriam, Qualitative Research and Case Study Applications in Education. San Francisco: Jossey-Bass Publishers, 1998. C. Conrad, A. Neumann, J. G. Haworth, and P. Scott, “Qualitative Research in Higher Education: Experiencing Alternative Perspectives and Approaches.” Needham Heights, MA: Ginn Press, 1993. R. J. Menges and A. E. Austin, “Teaching in higher education,” in Handbook of Research on Teaching, V. Richardson, Ed.: AERA, in press. L. Cuban, “Curriculum stability and change,” in The Handbook of Curriculum, P. Jackson, Ed., pp. 216-247. K. Forbus, “Using Qualitative Physics to Create Articulate Educational Software,” IEEE Expert, vol. May/June, pp. 32-41, 1997. J. Baher, “How Articulate Virtual Labs Can Help in Thermodynamics Education: A Multiple Case Study,” presented at Frontiers in Education 1998 Conference, Tempe, AZ, 1998. J. Ma, “A Case Study of Student Reasoning About Feedback Control in a Computer-Based Learning Environment,” presented at Frontiers in Education Conference, San Juan, Puerto Rico, 1999 Y. S. Lincoln and E. G. Guba, Naturalistic Inquiry. Newbury Park: SAGE Publications, 1985.

0-7803-5643-8/99/$10.00 © 1999 IEEE November 10 - 13, 1999 San Juan, Puerto Rico 29th ASEE/IEEE Frontiers in Education Conference 13c3-17