A COMPLETE UNDERSTANDING OF DISORIENTATION PROBLEMS IN WEB-BASED LEARNING

TOJET: The Turkish Online Journal of Educational Technology – July 2012, volume 11 Issue 3   A COMPLETE UNDERSTANDING OF DISORIENTATION PROBLEMS IN ...
17 downloads 0 Views 342KB Size
TOJET: The Turkish Online Journal of Educational Technology – July 2012, volume 11 Issue 3

 

A COMPLETE UNDERSTANDING OF DISORIENTATION PROBLEMS IN WEB-BASED LEARNING Yu-Cheng Shih Graduate Institute of Network Learning Technology National Central University [email protected] Pei-Ren Huang Graduate Institute of Network Learning Technology National Central University [email protected] Yung-Chi Hsu Graduate Institute of Network Learning Technology National Central University [email protected] Sherry Y. Chen (Corresponding author) Graduate Institute of Network Learning Technology National Central University & Department of Information Systems and Computing Brunel University United Kingdom [email protected] ABSTRACT Disorientation problems influence student learning. To address this issue, this study uses an integrative approach to investigate the causes and consequences of disorientation problems so that a complete understanding can be obtained. Unlike previous empirical studies, which use statistical techniques, this study attempts to expose unexpected relationships with a data mining approach. The results indicate that the causes of disorientation problems are associated with learners’ computer experience, the frequencies and time of using each navigation tool. On the other hand, the consequences of disorientation problems are concerned with learning effort and learning performance. Based on the findings, a framework is proposed to provide guidelines for the development of Web-based learning systems that are able to reduce learners’ disorientation problems. Keywords: disorientation, data mining, learning performance, computer experience, navigation behavior. 1. INTRODUCTION The Internet is presently being used as an information source for educational purposes (Tutkun, 2011). Thus, Web-based learning systems have become increasingly popular in past several years (Liu, Kantarcioglu and Thuraisingham, 2008). The major advantage of Web-based learning systems lies within their flexibility. Due to this flexibility, many learning platforms have been moved to the Web. This is possibly because Web-based learning systems utilize hypermedia structure, which provides a nonlinear way to help each individual develop his/her own paths (Deursen and Dijk, 2010). However, the nonlinear way may increase complexity so learners may feel confused to interact with the Web-based learning systems. One of the major reasons is that the nonlinear ways provide learners with multiple directions to reach their learning objectives. Nevertheless, not all of learners can identify a right direction so they may feel difficult to choose a right direction and experience disorientation problems (Amadieu et al., 2010). Disorientation problems negatively affect student learning in various ways. One of the negative effects is that learners cannot identify where they have been, where they are, and where they can go (Zhang and Wang, 2010). In other words, learners may feel uncomfortable finding right directions for themselves (Saadé and Otrakji, 2007). In addition, disorientation problems may hinder learners in reaching their desired learning objectives. Thus, good performance cannot be achieved which may, in turn, cause learners to lose their motivation and confidence while using Web-based learning systems. In the end, learners may also reject the use of Web-based learning systems in the future because of these losses (Demirbilek, 2009). Due to the importance of these problems, research into this issue has mushroomed lately. A number of studies have investigated why learners encounter disorientation problems. For example, some studies claimed that Copyright © The Turkish Online Journal of Educational Technology 1

 

TOJET: The Turkish Online Journal of Educational Technology – July 2012, volume 11 Issue 3

  human factors, such as prior knowledge (Mishra and Yadav, 2006) and gender differences (Van Oostendorp and Juvina, 2007), may greatly influence the levels of learners’ disorientation problems. Moreover, their navigation behavior may be yet another influential factor (Otter and Johnson, 2000). Other studies have indicated that learners’ disorientation problems may also have great effects on their learning performance (Ahuja and Webster, 2001). For instance, Zhang and Wang (2010) indicated that learners who are confused while using such systems cannot achieve learning objectives effectively and demonstrate low performance. In summary, learners’ disorientation problems are a complex topic related to multiple issues. Therefore, there is a need to conduct further investigation to clarify these issues. To this end, the study presented in this paper aims to provide a complete understanding of learners’ disorientation problems. To effectively reach our aim, a data mining approach is adopted in this study. This is because data mining can discover hidden relationships (Hand et al., 2001). In brief, this study employs a data mining approach to investigate why students have disorientation problems and how disorientation problems affect their learning. This paper is organized as follows. In Section 2, theoretical background is described. Section 3 then moves to describe methodology used to achieve our aims. Subsequently, the results related to the causes and consequences of disorientation problems are presented in Section 4. The paper then proceeds to Section 5, which proposes a framework based on our results. Finally, conclusions are drawn and future work is identified in Section 6. 2. THEORETICAL BACKGROUND 2.1. Disorientation Problems Previous studies indicate that learners may get lost or become disorientated when using the Web-based learning systems (Nielsen and Tahir, 2002). To face this problem, recently, a number of studies investigate why learners experience disorientation problems. Their findings include navigation behavior, prior knowledge and gender differences. With respect to navigation behavior, Otter and Johnson (2000) found that learners’ disorientation problems might become severe when they feel confused with which links they should choose and which navigation tools can help them locate relevant information. Moreover, an earlier study by Smith (1996) indicated that learners felt confused because the links they selected could not help them locate relevant information. Moreover, the search tool they used could not help them find information they want. These findings echoes a claim made by Liang and Sedig (2009), which indicated that the investigation of disorientation problems should take into account the design of navigation tools. With respect to prior knowledge, Mishra and Yadav (2006) found that different levels of learners’ prior knowledge can cause different levels of disorientation problems. As shown in their work, learners with the low level of prior knowledge can frequently experience disorientation problems while those with the high level of prior knowledge can easily avoid disorientation problems. An earlier study by Last et al. (2001) found that students with high prior knowledge were better able to navigate easily, remember where they had been, and decide how to get to where they wanted to go. Conversely, the students with low prior knowledge often suffered from disorientation, not knowing where they had been, or where they could go to find the information that they needed. With respect to gender differences, previous research shows that females and males experience different levels of disorientation problems. Ford and Miller (1996) found that males were less disoriented than females and females showed fewer understandings of how to use navigation tools. The other study by Ford, Miller and Moss (2001) indicated that females had difficulties in finding their ways effectively around the Internet and they were more likely to get lost and did not feel in control. The aforementioned studies indicated that several issues cause the disorientation problems. In addition, recent research also examined the consequences of disorientation problems. Some studies investigated the impacts of disorientation problems on learning performance. For instance, Amadieu et al. (2010) found that learners’ disorientation problems may influence their learning performance. More specifically, learners with serious disorientation problems demonstrate low performance while those without disorientation problems show high performance. This may be due to the fact that the former can easily lose the path of the subject content while the latter can freely locate information to follow the path of the subject contents. Likewise, the other study proposed by Ahuja and Webster (2001) indicated that learners with disorientation problems demonstrated low performance because they felt difficult to reflect what they had learnt and to locate relevant information they wanted. To this end, they failed to integrate the concepts they learned so it was difficult for them to improve their learning performance. The aforesaid studies explored the causes and consequences of learners’ disorientation problems independently. In other words, there is a lack of integrative studies that do comprehensive investigation to examine the causes and consequences of disorientation problems together. To this end, this study addresses this issue. Such Copyright © The Turkish Online Journal of Educational Technology 2

 

TOJET: The Turkish Online Journal of Educational Technology – July 2012, volume 11 Issue 3

  investigation includes multiple aspects. Thus, there is a need to use intelligent techniques, such as data mining, to conduct data analyses so that a global understanding of disorientation problems can be obtained. 2.2. Data Mining Data mining, also known as knowledge discovery (Fayyad and Uthurusamy, 1996), is an interdisciplinary area that encompasses techniques from a number of fields, including information technology, statistical analyses, and mathematic science (Bohen et al. 2003). A major function of data mining is the search for valuable information within large volumes of data (Hand et al., 2001). It can then be used to predict, model or identify interrelationships within a set of data (Urtubia et al., 2007) without a need to predefine underlying relationships between dependent and independent variables (Chang and Chen, 2005) as some of traditional statistical methods require. As opposed to the traditional statistical methods, data mining uses the data itself to uncover relationships and patterns. In doing so, hidden relationships, patterns, and interdependencies can be discovered, predictive rules can be generated, and interesting hypotheses can be found. These are the advantages of data mining (Hedberg, 1995; Gargano and Ragged, 1999). Due to such advantages, Data mining has been successfully applied in various fields, such as image retrieval (Park, Seo, and Jang, 2005), personalized applications (Chang, Chen, Chiu, and Chen, 2009), and electronic commerce (Liao, Ho, and Yang, 2009). These fields use a variety of data mining techniques, but they may be divided into two major categories: supervized learning and unsupervized learning. The former, which is also known as classification, refers to assigning objects to predefined categories or classes (Hastie, Tibshirani and Friedman, 2001). On the contrary, the latter, which is also known as clustering, is concerned with the division of data into groups of similar objects (Chen and Liu, 2008; Nolan, 2002). Classification refers to the data mining problem of attempting to discover predictive patterns where a predicted attribute is nominal or categorical. The predicted attribute is called the class. Subsequently, a data item is assigned to one of a predefined set of classes by examining its attributes (Changchien and Lu, 2001). In other words, the objective of classification is not to explore the data to discover interesting segments, but rather to decide how new items should be classified. For example, Esposito, Licchelli, and Semeraro (2004) built student models for an e-learning system based on the students’ level of performance: good, sufficient or insufficient. Clustering is concerned with the division of data into groups of similar objects. Each group, called a cluster, consists of objects that are similar between themselves and dissimilar to objects of other groups. For example, a study by Kim (2007) adopted a clustering algorithm to extract sequences with similar behavioral patterns. Their results indicated that learners’ navigation patterns are distinctive among different clusters. It implies that clustering can provide structural properties to identify dissimilarities between each group. Both clustering and classification are useful techniques but a problem of using classification is that there is a need to have a predicted attributes. If the predicted attribute was not properly selected, the accuracy of the results might be affected. Therefore, we choose to use clustering in our study because disorientation problems are affected by multiple aspects and also have multiple effects. 3. METHODOLOGY DESIGN To investigate the causes and consequences of learners’ disorientation problems, an empirical study was conducted at a university in Taiwan. This section describes the methodology design of the empirical study, including participants, the design of a Web-based learning system, procedures and data analyses. 3.1. Participants 50 students voluntarily took part in our study. To recruit these participants, a request was issued to students in lectures and further by email, making clear the nature of the study and their participation. Participants had diverse levels of Internet skills, background knowledge and computer experience so that we can identify the causes of disorientation problems from the perspective of human factors. 3.2. Web-based learning system To be able to measure students’ attitudes to web-based learning, they should have experience in using it (Usta, 2011). Thus, we developed a Web-based learning system, which introduces the principles of “Interaction design” and includes eight sections. The user interface consists of (a) a title bar used to present the subject of the Web-based learning system, (b) a navigation-tool panel used to help learners locate information from the Web-based learning system, and (c) the main body of the Web-based learning system, which provides the details Copyright © The Turkish Online Journal of Educational Technology 3

 

TOJET: The Turkish Online Journal of Educational Technology – July 2012, volume 11 Issue 3

  of the subject content. Figure 1 shows the layout of this Web-based learning system.

Figure 1. The Web-based learning system. The Web-based learning system provides multiple navigation tools, which include a main menu, keyword search, hierarchical map and alphabetical index. The keyword search is to allow learners to locate information for particular concepts with a search box. The hierarchical map uses a graphic to represent relationships among various topics. The alphabetic index is to list all of topics in an alphabetic order (Khalifa and Kwok, 1999; Nilsson and Mayer, 2002; Chen and Macredie, 2004). In other words, these navigation tools serve different purposes so learners are allowed to have freedom to develop their own navigation strategies. By doing so, the relationships between learners’ disorientation problems and their navigation strategies can be discovered. 3.3. Procedure The procedure consists of five steps (Figure 2). The details of each step are shown below.

Figure 2. Experimental procedure. (1)

(2)

(3)

All of the participants were required to fill out their personal information, e.g., the levels of participants’ subject knowledge and system experience. Such personal information was stored in a log file so that the relationships between human factors and disorientation problems can be analyzed. The participants were required to take a pre-test, which aims to objectively assess their prior knowledge of the subject content. The pre-test included 20 multiple-choice questions, each of which includes four possible responses: three different answers and an “I don’t know” option. The participants were required to interact with the Web-based learning system. During this period, their Copyright © The Turkish Online Journal of Educational Technology 4

 

TOJET: The Turkish Online Journal of Educational Technology – July 2012, volume 11 Issue 3

 

(4)

(5)

interaction with the Web-based learning system was recorded in a log file, including their choices of navigation tools and the time spent for the Web-based learning system. At the same time, the participants were also required to complete practical tasks, which could maintain learners’ motivation (Scanlon, 2000). Furthermore, performing these tasks is to offer the opportunities of experiencing interface features provided by the Web-based learning system. Thus, we were able to identify how learners perceived the Web-based learning system. After finishing interacting with the Web-based learning system, the participants were required to take the post-test to identify how much they had learnt from the Web-based learning system. The post-test was as same as the pre-test, which includes 20 multiple-choice questions, each with three different answers and an “I don’t know” option. The questions were matched on the pre-test and post-test so that each question on the pre-test had a corresponding similar (but not the same) question on the post-test. Creating similar questions was achieved by re-writing the question A questionnaire has the potential to collect cognitive and affective data quickly and easily (Kinshuk, 1996). At last, the participants, thus, needed to fill out a questionnaire to identify disorientation problems that learners experienced. The questionnaire was developed by an analysis of previous studies on disorientation problems (e.g., Saadé and Otrakji, 2007; Ahuja and Webster, 2001). There were 16 questions and each one used a five-point Likert scale ranging very much, quite a lot, average, not much, and not at all.

3.4. Data Analysis The aim of this study is to investigate the causes and consequences of learners’ disorientation problems. In order to reach our aim, learners’ responses obtained from the questionnaire are worked as attributes to create clusters. Among a variety of clustering techniques, K-means algorithm is widely used to partition the data into several clusters according to their similarities (Han and Kamber, 2001). In particular, it is frequently applied to analyze Web usage data. For example, a study by Wiwattanacharoenchai and Srivihok (2003) used K-means to create customer clusters from Web logs of various Internet banking Web sites. Their results showed that there was a clear distinction between the clusters, in terms of customer behavior. The major principle of the K-means algorithm uses k initial centers, each of which is assigned to each cluster, to partition data into k clusters. Each pattern in the cluster is decided based on the nearest distance between the pattern and each cluster center. However, a major limitation of using the K-means algorithm is that the number of clusters needs to be predefined. In other words, there is a need to identify the most suitable number of clusters to perform the K-means algorithm. Such an issue can be treated as parameter exploration (De Jong, 1975), which is used to decide the suitable value of parameters. The parameter exploration is useful when a dataset is not large. Thus, the K-means algorithm is suitable for this study because the dataset was not large. Therefore, the parameter exploration was applied to decide the parameters of the K-means algorithm in this study. Subsequently, the number of clusters is set for the large range of value to investigate the robustness of the clustering results. The suitable number of clusters is determined based on not only the smallest distance between the features in a same cluster, but also the largest distance between the features in different clusters. After doing so, we found that the K-means algorithm produces more efficient outcomes for three clusters. Subsequently, we select features to examine the corresponding characteristics of each cluster based on a comprehensive review by Chen and Macredie (2010). The details of these features are described in Table 1. The detailed differences among the three clusters were illustrated with the mean and standard deviation of each feature. In addition, the correlation analysis is also conducted to provide additional evidence as to the relationships among computer experience, disorientation problem, and learning performance. Table 1. The features used to examine each cluster. Features

Explanation

Human Factors

Navigation Behavior

Learning Effort

y Computer experience y Gender Differences y Frequencies of using navigation tools: the total number of times each navigational tool used y Time of using each navigation tool: the total amount of time each navigational tool used y Total Time spent for reading pages in the Web-based learning system

Copyright © The Turkish Online Journal of Educational Technology 5

 

TOJET: The Turkish Online Journal of Educational Technology – July 2012, volume 11 Issue 3

  Learning performance

y Post test score: the score obtained from the post-test y Gain Score: the difference between the post-test score and pre-test score

4. RESULTS AND DISCUSSION Three clusters were created with a K-means algorithm. As shown in Table 2, the percentage of learners within each cluster is satisfactory because the number of the members in each cluster is reasonably balanced. The mean and standard deviation (STD) of the responses to the questionnaire for each cluster are shown in Table 2. The high mean value represents the high level of disorientation problems that learners experience and vice versa. The tendency of each cluster is shown below.

Cluster (C) C1 C2 C3 y

y y

Table 2. Score from questionnaire for Each Cluster. Reponses to the questionnaire Mean STD 2.33 0.46 3.35 0.36 5.23 0.47

Instances 21(42%) 17(34%) 12(24%)

Cluster 1 (C1, N=21): Learners do not experience any difficulties while using the Web-based learning system. In other words, the learners in Cluster 1 clearly understand the meanings of buttons and icons. However, it would be helpful for them if a suggested route could be given through the Web-based learning system. Cluster 2 (C2, N=17): Learners are somewhat confused while using the Web-based learning system. Because the uses of the back/forward buttons are unclear to them. Consequently, it is difficult for these learners to keep track of topics which have been learnt. Cluster 3 (C3, N=12): Learners may be easily confused while using the Web-based learning system. The learners in Cluster 3 do not fully understand the meaning of buttons and icons and it is difficult to keep track of topics which they have learnt. Moreover, they also think that too many options are provided so they do not know how to choose a proper navigation tool to help them locate information.

The aforementioned results indicate that learners encounter different levels of disorientation problems. Some may feel comfortable with the Web-based learning system while others may feel very confused with the system. Thus, there is a need to further develop a complete understanding for this issue. To this end, it is necessary to investigate why they have disorientation problems and how disorientation problems influence their learning. In other words, we need to identify the causes and consequences of learners’ disorientation problems, which are presented in subsections below. 4.1. Causes of disorientation problems The results indicate that disorientation problems are caused by the levels of computer experience, the use of navigation tools, and time spent reading pages. The details are shown below. 4.1.1 Computer experience The levels of learners’ computer experience in each cluster are illustrated in Figure 3, which shows there is an inverse relationship between learners’ computer experience and their disorientation problems (r=-.275; p

Suggest Documents