International Journal of Automation and Computing

7(3), August 2010, 372-380 DOI: 10.1007/s11633-010-0517-5

Improving Decision Tree Performance by Exception Handling Appavu Alias Balamurugan Subramanian1 1

S. Pramala1

B. Rajalakshmi1

Ramasamy Rajaram2

Department of Information Technology, Thiagarajar College of Engineering, Madurai, India 2

Department of Computer Science, Thiagarajar College of Engineering, Madurai, India

Abstract: This paper focuses on improving decision tree induction algorithms when a kind of tie appears during the rule generation procedure for specific training datasets. The tie occurs when there are equal proportions of the target class outcome in the leaf node0 s records that leads to a situation where majority voting cannot be applied. To solve the above mentioned exception, we propose to base the prediction of the result on the naive Bayes (NB) estimate, k-nearest neighbour (k-NN) and association rule mining (ARM). The other features used for splitting the parent nodes are also taken into consideration. Keywords: Data mining, classification, decision tree, majority voting, naive Bayes (NB), k-nearest-neighbour (k-NN), association rule mining (ARM).

1

Introduction

The process of knowledge discovery in databases (KDD) is defined by Fayyad et al.[1] as “the non trivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data”. Data mining is the core step of the KDD process, which is concerned with a computationally efficient enumeration of patterns present in a database. Classification is a primary data mining task aimed at learning a function that classifies a database record into one of the several predefined classes based on the values of the record attributes[2] . The decision tree is an important classification tool and its various improvements such as ID3[3, 4] , ID4[5] , ID5[6] , ITI[7] , C4.5[8] , and CART[9] over the original decision tree algorithm have been proposed. All of them deal with the concept of incrementally building a decision tree in real time. In decision-tree learning, a decision tree is induced from a set of labelled training instances represented by a tuple of attribute values and a class label. Because of the vast search space, decision-tree learning is typically a greedy, top-down recursive process starting with the entire training data and an empty tree. An attribute that best partitions the training data is chosen as the splitting attribute for the root, and the training data are then partitioned into disjoint subsets satisfying the values of the splitting attribute. For each subset, the algorithm proceeds recursively until all instances in a subset belong to the same class. However, prior decision tree algorithms do not handle the exception, namely, “when two or more classes have equal probabilities in a tree leaf”. This paper investigates exception handling in decision tree construction. We examine few exceptions such as how to produce classifications from a leaf node which contains ties. In such a leaf, each class is represented equally, preventing the tree from using “majority voting” to output a classification prediction. We introduce new techniques for handling these exceptions and use real world datasets to show that these techniques improve clasManuscript received September 3, 2008; revised June 26, 2009

sification accuracy. This paper is organized as follows. Related works are stated in Section 2. Problem statements are illustrated in Section 3. Sections 4 and 5 aim at handling several exceptions occurring during decision tree induction. We compare the proposed algorithm to the most common algorithm of decision tree construction and evaluate the performance of our algorithm on a variety of standard benchmark datasets in Section 6. Section 7 concludes the paper besides presenting a number of issues for future research.

2

Related work

Decision-tree induction algorithm is one of the most successful learning algorithms, due to its various attractive features: simplicity, comprehensibility, absence of parameters, and the applicability to handle mixed-type data. Decision tree learning is one of the most widely used and practical methods for inductive learning. The ID3 algorithm[3] is a useful concept-learning algorithm because it can efficiently construct a well generalized decision tree. For nonincremental learning tasks, this algorithm is often an ideal choice for building a classification rule. However, for incremental learning tasks, it would be far preferable to accept instances incrementally, without the necessity to build a new decision tree each time. There exist several techniques to construct incremental decision tree based models. Some of the earlier efforts include ID4[5] , ID5[6] , ID5R[10] , and ITI[7] . All these systems work by using the ID3 style “information gain” measure to select the attributes. They are all designed to incrementally build a decision tree using one training instance at a time by keeping the necessary statistics (measure for information gain) at each decision node. Many learning tasks are incremental as new instances, or details become available over time. Decision tree induction algorithm works by building a tree, and updating it as new instances become available. The ID3 algorithm[3] can be

373

A. A. B. Subramanian et al. / Improving Decision Tree Performance by Exception Handling

used to learn incrementally by adding each new instance to the training set as it becomes available and by re-running ID3 against the enlarged training set. This is however computationally inefficient. The ID5[6] and ID5R[10] are both incremental decision tree builders that overcome the deficiencies of ID4. The essential difference is that when tree restructuring is required, instead of discarding a sub-tree due to its high entropy, the attribute that is to be placed at the node is pulled up and the tree structure below the node is retained. In the case of ID5[6] , the sub-trees are not recursively updated while in ID5R[10] they are updated recursively. Leaving the sub-trees un-restructured is computationally more efficient. However, the resulting sub-tree is not guaranteed to be the same as the one that would be produced by ID3 on the same training instances. The incremental tree inducer (ITI)[7] is a programme that constructs decision tree automatically from labelled examples. The most useful aspect of the ITI algorithm is that it provides a mechanism for incremental tree induction. If one has already constructed a tree, and then obtains a new labelled example, it is possible to present it to the algorithm, and have the algorithm revise the tree as necessary. The alternative would be to build a new tree from scratch, based on the augmented set of labelled examples, which is typically much more expensive. ITI handles symbolic variables, numeric variables, and missing data values. It includes a virtual pruning mechanism too. The development of decision tree learning leads to and is encouraged by a growing number of commercial systems such as C5.0/See5 (RuleQuest Research), MineSet (SGI), and Intelligent Miner (IBM). Numerous techniques have been developed to speed up decision-tree learning, such as designing a fast tree-growing algorithm, parallelization, and data partitioning. A number of strategies for decision tree improvements have been proposed in the literature [11 − 19]. They aim at “tweaking” an already robust model despite its main and obvious limitation. A number of ensemble classifiers have been proposed in the literature [20−24] which seem to have little improvement on accuracy especially when the added complexity of the method is considered.

3

noted as |A|. For each attribute ai , the set of possible values is denoted as Vi . The individual values are indicated by vij , where j is between 1 and the number of values for attribute ai , denoted as |Vi |. The notations used to represent the features of training data are, Aj , where j = 1, · · · , n for attributes, C for class attributes and the values for each attribute is Vij, where i = 1, · · · , n and j refers to the attribute to which it belongs. The decision tree induction algorithm has the chance of handling three different types of training data sets. The general structure of the training data set is shown in Table 1. Table 1

Structure of training set

A1

A2

A3

A4

A5

···

Am

C

V11

V12

V13

V14

V15

···

V1m

C1

V21

V22

V23

V24

V25

···

V2m

C2

V31

V32

V33

V34

V35

···

V3m

C3

···

···

···

···

···

···

···

···

Vn1

Vn2

Vn3

Vn4

Vn5

···

Vnm

Cn

After analyzing the decision tree algorithms, it has been found that the concept of majority voting has to handle different types of inputs. Consider the attribute Ak between A1 to Am , and C be the class attribute. The values of Ak are V1k and V2k , and let the values of the class attribute be C1 and C2 . Classification can be done if 1) Both Ak and C have only one distinct value; 2) When most of the attribute values V1k , V2k of particular attribute Ak belong to the same class is called as majority voting. The maximum occurrence of the distinct value and its corresponding distinct class label value is obtained from Table 2, which shows that majority voting is successful. Consider the training data set shown in Table 3 where majority voting fails. Table 2

Input format for majority voting condition Ak

C

V1k

C2

V2k

C2

V1k

C2

V1k

C1

Problem statements and overview If AK = V1k then C = C2

This paper studies one type of tie in the decision tree. The tie occurs when there are the same amount of different class cases in a leaf node. For the problem, the association rule mining (ARM), k-nearest neighbor (k-NN), and naive Bayes (NB) are integrated into the decision tree algorithm. ARM, k-NN, and NB are used to make prediction on the instances in the tie leaf node. WEKA[25, 26] uses a weak kind of tie breaking, such as taking the first value mentioned in the value declaration part of the WEKA data file, while the proposed solutions try to break ties more intelligently, such as to opt for using ARM, k-NN and NB. We use ARM, kNN, and NB to select the class value when majority voting does not work. The set of attributes used to describe an instance is denoted by A, and the individual attributes are indicated as ai , where i is between 1 and the number of attributes, de-

If AK = V2k then C = C2 . Table 3

Example training set where majority voting cannot be applied Ak

C

V1k

C1

V2k

C2

V1k

C1

V1k

C2

V1k

C2

Under this condition, only one rule can be generated as If Ak = V2k then C = C2 .

374

International Journal of Automation and Computing 7(3), August 2010

The count of the distinct value is 2 in Table 3. Hence, two rules should be generated from the table but only one rule is generated with the help of majority voting and the other rule cannot be generated If Ak = V1k then C = ? The value for Ak can be found by majority voting as Ak = V1k but its corresponding class value cannot be determined because among the four records there is an equal partition of class attribute C = C1 or C2 , in Table 3. Hence, the majority voting cannot be applied in this case. The decision tree induction algorithm does not give any specific solution to handle this problem.

4

The training tuples in Hayes-Roth dataset is given as input to decision tree induction algorithm. For the given training data set, the decision tree induction algorithm starts its recursive function of identifying the attribute with the highest information gain. On the first recursion, the attribute “age” gets the highest information gain and it divides the data set into two halves, which is the core idea behind the algorithm. The recursive function identifies the attributes at each iteration, and the rules are generated. The majority voting is the stopping condition for the recursive partitioning, and all possible data at that particular iteration is given in Table 4. The possible set of rules generated is shown in Tables 5 and 6. Table 4

Ties between classes

This paper addresses the problem of tree induction for supervised learning. We have proposed a method to handle the situation where the number of instances for different classes is the same at a leaf node. The basic algorithm for decision tree induction is a greedy algorithm that constructs decision trees in a top-down recursive divide-and-conquer manner. We have chosen a training data from the “Hayes-Roth database”. (The data are adapted from [27]. The class attribute, has three distinct values (namely, {1, 2, 3}); therefore, there are three distinct classes (m = 3). Let class C1 correspond to “1”, class C2 correspond to “2”, and class C3 correspond to “3”. There are 21 samples with class label “1” , 21 samples with class label “2”, and 19 samples with class label “3”. The expected information needed to classify a given sample is first derived to compute the information gain of each attribute. 21 21 ) log2 ( )− 59 59 21 21 19 19 ( ) log2 ( ) − ( ) log2 ( ) = 1.5833. 59 59 59 59

I(s1 , s2 ,s3 ) = I(21, 21, 19) = −(

Next, we need to compute the entropy of each attribute. In the attribute “hobby”. The distribution of “1”, “2”, and “3” samples for each value of “hobby” needs to be looked at, and the expected information gain for each of these distributions is computed. Table 5 Number

A sample partitioned dataset

Education level

Hobby

Class

3

1

2

4

3

3

1

1

1

2

1

2

3

2

2

1

2

1

2

2

2

3

3

2

3

1

1

1

3

1

2

3

2

3

2

1

3

3

1

In Table 4, “education level” gets the highest information gain, and hence it splits the table into four as “Education level = 3”, “Education level = 4”, “Education level = 2”, and “Education level = 1”, then the recursive function is applied on the table. In Table 7, both information gain and majority voting cannot be applied and hence decision tree induction algorithm chooses the class label value arbitrarily. Thus, for the same set of training samples, different rules are generated. This type of misconception occurs when recursive partitioning faces a data set which has an equal number of records, but the concept of majority voting works correctly only when the number of records is unequal. Fig. 1 shows majority voting failure with Hayes-Roth dataset.

Generated rules for Hayes-Roth dataset with “1” as the first class value

Generated rules

1

If Age = 1 and Marital status = 1 and Education level = 2, then Class = 1

2

If Age = 1 and Marital status = 1, and Education level = 4 and Hobby=3, then Class = 3

3

If Age = 1 and Marital status = 2 and Education level = 1, then Class = 1

4

If Age = 1 and Marital status = 2 and Education level = 2, then Class = 2

5

If Age = 1 and Marital status = 2 and Education level = 4 and Hobby=3, then Class = 1

6

If Age = 1 and Marital status = 2 and Education level = 3 and Hobby=3, then Class = 1

7

If Age = 1 and Marital status = 2 and Education level = 3 and Hobby=2, then Class = 2

8

If Age = 1 and Marital status = 2 and Education level = 3 and Hobby=1, then Class = 1

9

If Age = 1 and Marital status = 4, then Class = 3

10

If Age = 1 and Marital status = 3, then Class = 1

A. A. B. Subramanian et al. / Improving Decision Tree Performance by Exception Handling Table 6 Number

375

Generated rules for Hayes-ROTH dataset with “2” as the first class value

Generated rules

1

If Age = 1 and Marital status = 1 and Education level = 2, then Class = 1

2

If Age = 1 and Marital status = 1 and Education level = 4, and Hobby=3 then Class = 3

3

If Age = 1 and Marital status = 2 and Education level = 1, then Class = 1

4

If Age = 1 and Marital status = 2 and Education level = 2, then Class = 2

5

If Age = 1 and Marital status = 2 and Education level = 4 and Hobby=3, then Class = 1

6

If Age = 1 and Marital status = 2 and Education level = 3 and Hobby=3, then Class = 1

7

If Age = 1 and Marital status = 2 and Education level = 3 and Hobby=2, then Class = 2

8

If Age = 1 and Marital status = 2 and Education level = 3 and Hobby=1, then Class = 2

9

If Age = 1 and Marital status = 4, then Class = 3

10

If Age = 1 and Marital status = 3, then Class = 1

Fig. 1 The attribute age has the highest information gain and therefore becomes a test attribute at the root node of the decision tree. Branches are grown for each value of age. The samples are partitioned according to each branch Table 7

5

Example for a situation where majority voting is not possible Hobby

Class

1

2

2

2

3

2

1

1

2

1

3

1

Methods for handling ties between classes

This paper addresses the research question in the context of decision tree induction: which class to choose when, classified with respect to an attribute, the number of records having the different class values are equal, i.e., when majority voting fails. The NB probabilistic model, k-NN, and ARM are used to answer the problem.

5.1

Naive Bayes (NB)

In this paper, it has been proposed to use the NB classifier to select the class value when majority voting does not work. The presented approach deals with the shortcomings in information gain based decision tree induction

algorithm. In order to avoid the arbitrary selection of class label due to failure of the “majority voting”, the usage of NB to assign the class label is proposed. The class label chosen by the NB would be better than an arbitrary selection. Thus, decision tree naive Bayes (DTNB) can be called as the extension of decision tree algorithm which does not give contradictory rules. The traditional decision tree induction algorithm data set would fall under any one of these categories: 1) consistent data or 2) inconsistent data. However, with DTNB algorithm, any data set considered for classification would fall under any one of these categories: 1) consistent data identified by alpha rules, 2) inconsistent data identified by alpha rules, 3) consistent data identified by beta rules, or 4) inconsistent data identified by beta rules. Inconsistent data are those which are not classified in the given training data set. Hence, the efficiency of the algorithm decreases due to its inapplicability in classifying the given data. The records, which are considered as inconsistent, are identified by alpha rules. Noisy data may be present in the training data due to human errors, that is, the user records irrelevant values in the data set. The algorithm should be developed in such a way that it is capable of handling noisy data also. When a rule is formed by a set of records in which the value of the class label accounts for more than 50 % than

376

International Journal of Automation and Computing 7(3), August 2010

its counterpart, then that rule is acceptable and those instances that contradict this rule will be recorded as the noisy data. The inconsistent records are those which are identified as noise by beta rules. Consider the attribute “hobby” in Fig. 1, which cannot decide the class label0 s value. The proposed solution considers the path of the tree as the test data and all the remaining data as the training set for this problem. This dataset is given to the probability based algorithm (simple naive Bayesian) and the path of tree is considered as the test data, i.e., If Age = 1 and Marital status = 2 and Education level = 3 and Hobby = 1, then Class = ? NB is a classification technique[28−31] that is both predictive and descriptive. It analyzes the relationship between each independent and dependent variable and derives a conditional probability for each relationship. When a new case is analyzed, a prediction is made by combining the effects of the independent variables on the dependent variable (the outcome that is predicted). NB requires only one pass through the training set to generate a classification model, which makes it the most efficient data mining technique. Bayesian classification is based on Bayes theorem[29, 32] . Let X be a data sample whose label is unknown and let H be some hypothesis, such that the data sample X belongs to a specified class C. According to Bayes theorem, for classification problems, P (H|X) is to be determined, i.e., the probability that the hypothesis H holds true, given the observed data sample X. P (X), P (H), and P (X|H) may be estimated from the given data. Bayes theorem is useful in that it provides a way of calculating the posterior probability, P (H|X), from P (H), P (X), and P (X|H). From Bayes theorem, it follows: P (H|X) =

P (X|H)P (H) P (X)

(1)

where X = (Age = 1 and Marital status = 2 and Education level = 3 and Hobby = 1) NB aims at predicting the class value (Class = 1 or Class = 2), and NB does it probabilistically. We use the notation that Pr(A) denotes the probability of an event A, and Pr(A/B) denotes the probability of A conditional on another event B. The hypothesis H is that class will be “1” or “2”, and our aim to find Pr(H/E). E is evidence. The evidence E is the particular combination of attribute values, Age = 1 and Marital status = 2 and Education level = 3 and Hobby = 1. By maximizing P (X|Ci )P (Ci ), for i = 1, 2, P (Ci ), the prior probability of each class can be computed based on the training samples. Therefore, the naive Bayesian classifier predicts Class = 1 for sample X. The procedure for handling ties between classes using NB are given in Algorithm 1. Algorithm 1. Algorithm for handling ties between classes using NB 1) If attribute-list is empty then 2) If training data rule is null 3) Return N as a leaf node labeled with the most common class in samples // Majority voting This rule traversal from the root to the leaf is alpha rule. If (records satisfy alpha rule) i) Correctly classified (as normal decision tree induction algorithm)

Else ii) Incorrectly classified (as normal decision tree induction algorithm) 4) Else 5) Return N as a leaf node labelled with the attribute corresponding to class label value from probability based algorithm (naive Bayesian algorithm) This traversal of the rule from the root to the leaf is called the beta rule // A unique rule If (records satisfy beta rule) Correctly classified (These records cannot be handled by decision tree induction algorithm) Else Incorrectly classified.

Hence, the class value is assigned based on the given training data set where majority voting is not possible. The rules developed in this manner are called the beta rules as it has the unique feature of handling the exception due to failure of majority voting.

5.2

k-nearest neighbour (k-NN)

In this section, the usage of k-NN is proposed to overcome a few problems that result due to the failure of majority voting. This is an enhancement of the decision tree induction algorithm to deal with the cases of unpredictability. We explain the needs that drive the change, by describing specific problems of the algorithm, suggest solutions, and test them with benchmark data sets. If two or more classes have equal probabilities in a tree leaf, the assigning of class label using k-NN algorithm is proposed to solve the problem. k-NN is suitable for classification models[33−35] . Unlike other predictive algorithms, the training data is not scanned or processed to create the model. Instead, the training data is the model itself. When a new case or instance is presented to the model, the algorithm looks at the entire data to find a subset of cases that are most similar to it and uses them to predict the outcome. For distance measure, we use a simple metric that is computed by summing the scores for each of the independent columns. The procedure for handling ties between classes using instance based learning are given in Algorithm 2. Algorithm 2. Algorithm for handling ties between classes using k-NN 1) If attribute-list is empty then 2) If training data rule is null 3) return N as a leaf node labeled with the most common class in samples // Majority voting This rule traversal from the root to the leaf is alpha rule. If (records satisfy alpha rule) i) Correctly classified (as normal decision tree induction algorithm) Else ii) Incorrectly classified (as normal decision tree induction algorithm) 4) Else 5) Return N as a leaf node labelled with the attribute corresponding to class label value from instance based learning (k-NN algorithm) This traversal of the rule from the root to the leaf is called the beta rule // A unique rule If (records satisfy beta rule) Correctly classified (These records cannot be handled by decision tree induction algorithm) Else Incorrectly classified.

377

A. A. B. Subramanian et al. / Improving Decision Tree Performance by Exception Handling

In the traditional decision tree induction algorithm, the data set would fall under any one of the categories, as explained in the previous section. The proposed algorithm considers the path of the tree as the test data and all the remaining data as the training set for this problem. This training data set is given to the instance based learning (simple k-NN classifier) and path of tree is considered as the test data i.e., let X= (Age = 1 and Marital status = 2 and Education level = 3 and Hobby = 1). Output = y 0 = 1. If Age = 1 and Marital status = 2 and Education level = 3 and Hobby = 1, then Class = 1.

5.3

Association rule mining (ARM)

ARM finds interesting associations and correlation relationships among large sets of data items[36] . If two or more classes have equal probabilities in a tree leaf, the usage of ARM is proposed to assign the class label value. ARM can be used to solve the majority voting failure problem existing in decision tree induction algorithm. After finding out the rules, the class value for the conflicting data is decided as follows: 1) Find out the rule having the highest number of attributes. 2) If two or more rules have the highest number of attributes but different class values, then find out the number of rules for each different class values having the highest number of attributes and those class values are returned back as the appropriate class label. Otherwise, return the corresponding label of the rule having highest number of attributes. The procedure for handling ties between classes using ARM are given in Algorithm 3. Using ARM, we can predict that, if Hobby = 1 and Age = 1 and Educational level = 3 and Marital status = 2, then Class = 1. Algorithm 3. Algorithm for handling ties between classes using ARM Generate a decision tree from the training tuples of data D Input: Training data tuples D Output: Decision tree Procedure: 1) Create a node in the decision tree 2) If all tuples in the training data belong to the same class C, the node created above is a leaf node labeled with class C 3) If there are no remaining attributes on which the tuples may be further partitioned i) If any one class has higher majority count than others // Majority voting success Convert the node created above into a leaf node and label it with majority class C at that instance // Apply majority voting ii) Else // majority voting fails a) Apply ARM algorithm to find out the frequent pattern in training data D for conflicting rule b) Tune the conflicting rule depending on class value obtained from frequent patterns 4) Else Apply attribute selection method to find the “best” splitting-criterion 5) Label the created node with splitting criterion 6) Remove splitting attribute from the attribute list 7) For each outcome of splitting attribute, partition the tuples and grow sub trees for each partition 8) If number of tuples satisfying an outcome of the splitting attribute is empty Apply majority voting

9) Else Apply the above algorithm to those tuples that satisfy outcome of the splitting attribute.

6

Experimental results mance evaluation

and

perfor-

The performance of the proposed methods was evaluated on artificial and real-world domains. Artificial domains are useful because they allow variation in parameters, and aid us in understanding the specific problems that algorithms exhibit, and the test conjectures. Real world domains are useful because they come from real-world problems and are therefore actual problems on which we would like to improve performance. All real-world data sets used are from the University of California Irvine (UCI) repository[27] , which contains over 177 data sets mostly contributed by researchers in the field of machine learning. The data sets that possess majority voting problem, i.e., two records having the same attribute values but different class values, are purposely chosen for experimentation to prove the efficiency of our algorithm. Experiments are conducted to measure the performance of the proposed methods, and it was observed that these data sets are efficiently classified. Tables 8 and 9 give a description of the dataset used in the experiment. The proposed algorithms0 performance is compared with various existing decision tree induction algorithms. Tables 10 and 11 show, for each data set, the estimated predictive accuracy of the decision tree with k-NN, decision tree with ARM, and decision tree with NB versus other traditional decision tree methods. As one can see from Tables 10 and 11, the predictive accuracy of the DTARM, DT-KNN, and DT-NB tends to be better than the accuracy of traditional decision tree induction algorithms. There are four common approaches for estimating the accuracy such as using training data, using test data, crossvalidation, and percentage splitting[37] . The evaluation function used is percentage splitting and 10 fold cross validation. In Tables 10 and 11, datasets with the problem of majority voting are considered and their accuracy is predicatively compared with that of the traditional decision tree induction algorithm. The proposed algorithm has provided better results than C4.5 besides solving the exceptions caused due to the failure of majority voting. This paper presents an algorithm based on decision tree with k-NN, decision tree with ARM, and decision tree with NB which make use of the decision tree as the main structure and conditionally applies ARM, k-NN, and NB at some of the leaf nodes in order to handle cases where a node has equal probability for more than one class. The nearest neighbor technique used in DT-KNN algorithm has a large number of drawbacks when compared to the NB technique used in DT-NB algorithm. These drawbacks include a lack of descriptive output, reliance on an arbitrary distance measure, and the performance implications of scanning the entire data set to predict each new case. The NB method, on the other hand, has a number of qualities. It is the most efficient technique with respect to training and it can make predictions from partial data.

378

International Journal of Automation and Computing 7(3), August 2010

The biggest theoretical drawback with NB is the need for independence between the columns which is usually not a problem in practice. We can see the performance of each classifier, still, some conclusions can be made from this experiment: 1) Using NB, k-NN, and ARM to resolve tie such as two Table 8

or more classes have equal probabilities in a tree leaf, which is better than a random decision at the node). 2) Using NB to resolve tie such as two or more classes have equal probabilities in a tree leaf, which is better than an ARM and k-NN at the node. The accuracy across various classifiers as shown in Fig. 2.

The datasets that possess majority voting problem in the UCI repository

Dataset

Number of instances

Number of attributes

Abalone

4177

9

Associated task Classification

Pima-diabetes

768

9

Classification Classification

Flag

194

30

Glass

214

11

Classification

Housing

506

14

Classification

Image segmentation

210

20

Classification

Ionosphere

351

35

Classification

Iris

150

5

Classification

Liver disorder

345

7

Classification

Parkinson

195

24

Classification

Yeast

1484

10

Classification

Table 9

The datasets that possess majority voting problem in the UCI repository (after discretization)

Dataset

Number of instances

Number of attributes

Blood transfusion

748

5

Associated task Classification

Contraceptive method choice

1473

10

Classification Classification

Concrete

1030

9

Forest-fires

517

13

Classification

Haberman

306

4

Classification

Hayes-Roth

132

6

Classification

Solar flare 1

323

13

Classification

Solar flare 2

1066

13

Classification

Spect-heart

267

23

Classification

Teaching assistant evaluation

151

6

Classification

Table 10

Predictive accuracy of the proposed methods versus C4.5, NB, k-NN with percentage splitting

Data set

C4.5

NB

k-NN

Decision tree with ARM

Decision tree with NB

Decision tree with k-NN

Blood transfusion

84.61

86.26

83.51

85.16

90.11

87.36

Solar flare 1

67.9

56.79

70.37

85.18

91.46

88.89

Glass

66.04

28.3

69.81

75.47

86.79

81.13

Haberman

59.57

66.67

54.54

87.88

90.91

87.88

Hayes-Roth

67.5

72.5

57.5

85.00

92.50

87.50

Liver disorder

63.95

46.51

66.28

86.04

90.70

87.20

Parkinson

50

50

33.33

86.67

90.00

86.67

Teaching assistant evaluation

54

70.27

89.19

86.48

94.59

86.48

Mean

64.20

59.66

65.57

84.735

90.88

86.64

Table 11

Predictive accuracy of the proposed methods versus C4.5, NB, k-NN with ten-fold cross validation

Data set

C4.5

NB

k-NN

Decision tree with ARM

Decision tree with NB

Decision tree with k-NN

Solar flare 1

71.21

65.01

69.66

85.39

89.16

87.62

Hayes-Roth

72.73

80.30

57.58

89.23

93.94

91.67

Parkinson

80.51

69.23

89.74

97.89

98.97

97.89

Teaching assistant evaluation

52.98

50.33

62.25

86.00

93.38

89.40

Mean

69.36

66.22

69.81

89.63

93.86

91.65

A. A. B. Subramanian et al. / Improving Decision Tree Performance by Exception Handling

379

[7] P. E. Utgoff. An improved algorithm for incremental induction of decision trees. In Proceedings of the 11th International Conference on Machine Learning, pp. 318–325, 1994. [8] J. R. Quinlan. C4.5: Programs for Machine Learning, Morgan Kaufmann Publishers, 1993. [9] L. Breiman, J. H. Friedman, R. A. Olsen, C. J. Stone. Classification and Regression Trees, Wadsworth and Brooks, 1984. [10] P. E. Utgoff. Incremental induction of decision trees. Machine Learning, vol. 4, no. 2, pp. 161–186, 1989. [11] S. A. Balamurugan, R. Rajaram. Effective and efficient feature selection for large scale data using Bayes0 theorem. International Journal of Automation and Computing, vol. 6, no. 1, pp. 62–71, 2009. [12] W. Buntine. Learning classification trees. Statistics and Computing, vol. 2, no. 2, pp. 63 –73, 1992. [13] C. R. P. Hartmann, P. K. Varshney, K. G. Mehrotra, C. L. Gerberich. Application of information theory to the construction of efficient decision trees. IEEE Transactions on Information Theory, vol. 28, no. 4, pp. 565–577, 1982.

Fig. 2

7

Predictive accuracy on classifiers

Conclusions

This paper proposes enhancements to the basic decision tree induction algorithm. The enhancement handles the case where the numbers of instances for different classes are the same in a leaf node. The results on the UCI data sets show that the proposed modification procedure has better predictive accuracy over the existing methods in the literature. Future work includes the use of a more robust classifier instead of NB, k-NN, and ARM. Building a tree for each tied feature with lower cost is also an area that demands attention in the near future.

References [1] U. M. Fayyad, G. Piatetsky-Shapiro, P. Smyth. From data mining to knowledge discovery: An overview. Advances in Knowledge Discovery and Data Mining, U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, R. Uthurusamy, Menlo Park, CA, USA: American Association for Artificial Intelligence, pp. 1–34, 1996. [2] J. W. Han, M. Kamber. Data Mining: Concepts and Techniques, Morgan Kaufmann Publishers, 2006. [3] J. R. Quinlan. Induction of decision trees. Journal of Machine Learning, vol. 1, no. 1, pp. 81–106, 1986. [4] J. R. Quinlan. Simplifying decision trees. International Journal of Man-machine Studies, vol. 27, no. 3, pp. 221–234, 1987. [5] P. E. Utgoff. Improved training via incremental learning. In Proceedings of the 6th International Workshop on Machine Learning, Morgan Kaufmann Publishers Inc., Ithaca, New York, USA, pp. 362–365, 1989. [6] P. E. Utgoff. ID5: An incremental ID3. In Proceedings of the 5th International Conference on Machine Learning, Morgan Kaufmann Publishers Inc., Ann Arbor, MI, USA, pp. 107–120, 1988.

[14] J. Mickens, M. Szummer, D. Narayanan. Snitch interactive decision trees for troubleshooting misconfigurations. In Proceedings of the 2nd USENIX Workshop on Tackling Computer Systems Problems with Machine Learning Techniques, USENIX Association, Cambridge, MA, USA, Article No. 8, 2007. [15] R. Kohavi, C. Kunz. Option decision trees with majority votes. In Proceedings of the 14th International Conference on Machine Learning, Morgan Kaufmann, pp. 161–169, 1997. [16] R. Carina, A. Niculescu-Mizil. An empirical comparison of supervised learning algorithms. In Proceedings of the 23rd International Conference on Machine Learning, ACM, Pittsburgh, Pennsylvania, USA, pp. 161–168, 2006. [17] J. C. Schlimmer, D. Fisher. A case study of incremental concept induction. In Proceedings of the 5th National Conference on Artificial Intelligence, Morgan Kaufmann, Philadelpha, USA, pp. 496–501, 1986. [18] J. C. Schlimmer, R. Granger. Beyond incremental processing: Tracking concept drift. In Proceedings of the 5th National Conference on Artificial Intelligence, vol. 1, pp. 502– 507, 1986. [19] P. E. Utgoff, N. C. Berkman, J. A. Clouse. Decision tree induction based on efficient tree restructuring. Machine Learning, vol. 29, no. 1, pp. 5–44, 2004. [20] H. A. Chipman, E. I. George, R. E. McCulloch. Bayesian CART model search. Journal of the American Statistical Association, vol. 93, no. 443, pp. 935–948, 1998. [21] R. Kohavi. Scaling up the accuracy of naive Bayes classifiers: A decision tree hybrid. In Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining, AAAI Press, pp. 202–207, 1996. [22] L. M. Wang, S. M. Yuan, L. Li, H. J. Li. Improving the performance of decision tree: A hybrid approach. Conceptual Modeling, Lecture Notes in Computer Science, Springer, vol. 3288, pp. 327–335, 2004. [23] Y. Li, K. H. Ang, G. C. Y. Chong, W. Y. Feng, K. C. Tan, H. Kashiwagi. CAutoCSD-evolutionary search and optimisation enabled computer automated control system design. International Journal of Automation and Computing, vol. 1, no. 1, pp. 76–88, 2006. [24] Z. H. Zhou, Z. Q. Chen. Hybrid decision tree. Journal of Knowledge-based Systems, vol. 15, no. 8, pp. 515–528, 2002. [25] WEKA. Open Source Collection of Machine Learning Algorithm.

380

International Journal of Automation and Computing 7(3), August 2010

[26] I. H. Witten, E. Frank. Data Mining-practical Machine Learning Tools and Techniques with Java Implementation, 2nd Edition, 2004. [27] C. L. Blake, C. J. Merz. UCI Repository of Machine Learning Databases, [Online], Available: http:// www.ics.uci.edu/˜mlearn/mlrepository.html, 2008. [28] E. Frank, M. Hall, B. Pfahringer. Locally weighted naive Bayes. In Proceedings of Conference on Uncertainty in Artificial Intelligence, Morgan Kaufmann, pp. 249–256, 2003. [29] J. Joyce. Bayes Theorem, Stanford Encyclopedia of Philosophy, 2003. [30] L. Jiang, D. Wang, Z. Cai, X. Yan. Survey of improving naive Bayes for classification. In Proceedings of the 3rd International Conference on Advanced Data Mining and Applications, Springer, vol. 4632, pp. 134–145, 2007. [31] P. Langley, W. Iba, K. Thompson. An analysis of Bayesian classifiers. In Proceedings of the 10th National Conference on Artificial Intelligence, AAAI press and MIT press, pp. 223–228, 1992. [32] J. M. Bernardo, A. F. Smith. Bayesian Theory, John Wiley & Sons, 1993. [33] D. W. Aha, D. Kibler, M. K. Albert. Instance-based learning algorithms. Machine Learning, vol. 6, no. 1, pp. 37–66, 1991. [34] T. M. Cover, P. E. Hart. Nearest neighbour pattern classification. IEEE Transactions on Information Theory, vol. 13, no. 1, pp. 21–27, 1967. [35] S. M. Weiss. Small sample error rate estimation for knearest neighbour classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no. 3, pp. 285– 289, 1991. [36] T. Fukuda, Y. Morimoto, S Morishita, T. Tokuyama. Data mining with optimized two-dimensional association rules. ACM Transactions on Database Systems, vol. 26, no. 2, pp. 179–213, 2001. [37] R. Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pp. 1137–1143, 1995.

Appavu Alias Balamurugan Subramanian is a Ph. D. candidate in the Department of Information and Communication Engineering at Anna University, Chennai, India. He is also a faculty at Thiagarajar College of Engineering, Madurai, India. His research interests include data mining and text mining. E-mail: [email protected] (Corresponding author) S. Pramala received the B. Tech. degree in the Department of Information Technology at Thiagarajar College of Engineering, Madurai, India in 2009, and the B. Tech. degree in information technology in May, 2010. Her research interests include data mining, where classification and prediction methods dominate her domain area, mining of frequent patterns, associations and correlations existing in test data. E-mail: [email protected] B. Rajalakshmi is doing her undergraduate course in information technology. She qualifies as a final year student (2009) at Thiagarajar College of Engineering, Madurai, India. She received B. Tech. degree in information technology in May, 2010. Her research interests include textual data mining, usage of various classification techniques for efficient retrieval of data, data pruning, and pre-processing. E-mail: [email protected] Ramasamy Rajaram received the Ph. D. degree from Madurai Kamaraj University, India. He is a professor of Department of Computer Science and Information Technology at Thiagarajar College of Engineering, Madurai, India. His research interests include data mining and information security. E-mail: [email protected]