Information Retrieval and Data Mining Summer Semester 2015 TU Kaiserslautern Prof. Dr.-Ing. Sebastian Michel Databases and Information Systems Group (AG DBIS) http://dbis.informatik.uni-kl.de/ Information Retrieval and Data Mining, SoSe 2015, S. Michel

1

Chapter VI: Classification 1. Motivation and Definitions 2. Decision Trees 3. Bayes Classifier 4. Support Vector Machines (only as teaser)

Tan, Steinbach & Kumar, Chapter 8 Information Retrieval and Data Mining, SoSe 2015, S. Michel

2

1. Classification: Example Classifier age? youth student? no

no

middle_age?

senior

yes yes

yes

credit_rating? fair

no

excellent

yes

A decision tree for the concept buys_computer, indicating whether a customer at an electronic shop is likely to purchase a computer.

source: Han&Kamber

Information Retrieval and Data Mining, SoSe 2015, S. Michel

3

Classification: Definition • Given a collection of records (training set) – Each record contains a set of attributes, one of the attributes is the class.

© Tan,Steinbach, Kumar

• Find a model for class attribute as a function of the values of other attributes. • Goal: previously unseen records should be assigned a class as accurately as possible. – A test set is used to determine the accuracy of the model. Usually, the given data set is divided into training and test sets, with training set used to build the model and test set used to validate it. Information Retrieval and Data Mining, SoSe 2015, S. Michel

4

Examples of Classification Task • Predicting tumor cells as benign or malignant • Classifying credit card transactions as legitimate or fraudulent

© Tan,Steinbach, Kumar

• Categorizing news stories as finance, weather, entertainment, sports, etc • Classifying persons into tax evaders and tax payers. Information Retrieval and Data Mining, SoSe 2015, S. Michel

5

Illustrating Classification Task Tid

Attrib1

1

Yes

Large

125K

No

2

No

Medium

100K

No

3

No

Small

70K

No

4

Yes

Medium

120K

No

5

No

Large

95K

Yes

6

No

Medium

60K

No

7

Yes

Large

220K

No

8

No

Small

85K

Yes

9

No

Medium

75K

No

10

No

Small

90K

Yes

Attrib2

Attrib3

Class

Learning algorithm Induction Learn Model Model

10

© Tan,Steinbach, Kumar

Training Set Tid

Attrib1

Attrib2

Attrib3

11

No

Small

55K

?

12

Yes

Medium

80K

?

13

Yes

Large

110K

?

14

No

Small

95K

?

15

No

Large

67K

?

Apply Model

Class

Deduction

10

Test Set

Information Retrieval and Data Mining, SoSe 2015, S. Michel

6

Classification model evaluation • Much the same measures as with IR methods

Predicted class

– Focus on accuracy and error rate

Class = 1

Class = 0

Class = 1

f11

f10

Class = 0

f01

f00

– But also precision, recall, F-scores, … Information Retrieval and Data Mining, SoSe 2015, S. Michel

7

Overview Classification Techniques • • • • •

Decision-Tree-based Methods Rule-based Methods Naïve Bayes Support Vector Machines ……

Information Retrieval and Data Mining, SoSe 2015, S. Michel

8

© Tan,Steinbach, Kumar

Example of a Decision Tree Tid Refund Marital Status

Taxable Income Cheat

1

Yes

Single

125K

No

2

No

Married

100K

No

3

No

Single

70K

No

4

Yes

Married

120K

No

5

No

Divorced 95K

Yes

6

No

Married

No

7

Yes

Divorced 220K

No

8

No

Single

85K

Yes

9

No

Married

75K

No

10

No

Single

90K

Yes

60K

Splitting Attributes

Refund Yes

No

NO

MarSt Single, Divorced TaxInc

< 80K NO

Married NO

> 80K YES

10

Training Data

Model: Decision Tree

Information Retrieval and Data Mining, SoSe 2015, S. Michel

9

2. Decision Trees

© Tan,Steinbach, Kumar

MarSt Tid Refund Marital Status

Taxable Income Cheat

1

Yes

Single

125K

No

2

No

Married

100K

No

3

No

Single

70K

No

4

Yes

Married

120K

No

5

No

Divorced 95K

Yes

6

No

Married

No

7

Yes

Divorced 220K

No

8

No

Single

85K

Yes

9

No

Married

75K

No

10

No

Single

90K

Yes

60K

Married NO

Single, Divorced Refund

No

Yes NO

TaxInc

< 80K NO

> 80K YES

There could be more than one tree that fits the same data!

10

Information Retrieval and Data Mining, SoSe 2015, S. Michel

10

Decision Tree Classification Task Tid

Attrib1

1

Yes

Large

125K

No

2

No

Medium

100K

No

3

No

Small

70K

No

4

Yes

Medium

120K

No

5

No

Large

95K

Yes

6

No

Medium

60K

No

7

Yes

Large

220K

No

8

No

Small

85K

Yes

9

No

Medium

75K

No

10

No

Small

90K

Yes

Attrib2

Attrib3

Class

Tree Induction algorithm Induction Learn Model Model

10

© Tan,Steinbach, Kumar

Training Set

Tid

Attrib1

Attrib2

Attrib3

11

No

Small

55K

?

12

Yes

Medium

80K

?

13

Yes

Large

110K

?

14

No

Small

95K

?

15

No

Large

67K

?

Apply Model

Class

Decision Tree

Deduction

10

Test Set

Information Retrieval and Data Mining, SoSe 2015, S. Michel

11

Apply Model to Test Data Test Data

Start from the root of tree.

Refund

Taxable Income Cheat

No

80K

Married

?

10

Yes

No

NO

MarSt Single, Divorced TaxInc < 80K

© Tan,Steinbach, Kumar

Refund Marital Status

NO

Married

NO > 80K YES

Information Retrieval and Data Mining, SoSe 2015, S. Michel

12

Apply Model to Test Data Test Data

Refund

Taxable Income Cheat

No

80K

Married

?

10

Yes

No

NO

MarSt Single, Divorced TaxInc < 80K

© Tan,Steinbach, Kumar

Refund Marital Status

NO

Married

NO > 80K YES

Information Retrieval and Data Mining, SoSe 2015, S. Michel

13

Apply Model to Test Data Test Data

Refund

Taxable Income Cheat

No

80K

Married

?

10

Yes

No

NO

MarSt Single, Divorced TaxInc < 80K

© Tan,Steinbach, Kumar

Refund Marital Status

NO

Married

NO > 80K YES

Information Retrieval and Data Mining, SoSe 2015, S. Michel

14

Apply Model to Test Data Test Data

Refund

Taxable Income Cheat

No

80K

Married

?

10

Yes

No

NO

MarSt Single, Divorced TaxInc < 80K

© Tan,Steinbach, Kumar

Refund Marital Status

NO

Married

NO > 80K YES

Information Retrieval and Data Mining, SoSe 2015, S. Michel

15

Apply Model to Test Data Test Data

Refund

Taxable Income Cheat

No

80K

Married

?

10

Yes

No

NO

MarSt Single, Divorced TaxInc < 80K

© Tan,Steinbach, Kumar

Refund Marital Status

NO

Married

NO > 80K YES

Information Retrieval and Data Mining, SoSe 2015, S. Michel

16

Apply Model to Test Data Test Data

Refund

Taxable Income Cheat

No

80K

Married

?

10

Yes

No

NO

MarSt Single, Divorced TaxInc < 80K

© Tan,Steinbach, Kumar

Refund Marital Status

NO

Married

Assign Cheat to “No”

NO > 80K YES

Information Retrieval and Data Mining, SoSe 2015, S. Michel

17

Classifying a Record with a Decision Tree • Given a decision tree. • How to classify a test record? • Start at root note and apply the test condition to the record and follow the appropriate branch. • If this leads to internal node, again apply test condition and follow branch. • Otherwise, if at leave node, assign class of leave node to record. • Repeat until at leave node. Information Retrieval and Data Mining, SoSe 2015, S. Michel

18

Decision Tree Classification Task Tid

Attrib1

Attrib2

Attrib3

Class

1

Yes

Large

125K

No

2

No

Medium

100K

No

3

No

Small

70K

No

4

Yes

Medium

120K

No

5

No

Large

95K

Yes

6

No

Medium

60K

No

7

Yes

Large

220K

No

8

No

Small

85K

Yes

9

No

Medium

75K

No

10

No

Small

90K

Yes

Tree Induction algorithm Induction Learn Model Model

10

© Tan,Steinbach, Kumar

Training Set

Tid

Attrib1

Attrib2

Attrib3

11

No

Small

55K

?

12

Yes

Medium

80K

?

13

Yes

Large

110K

?

14

No

Small

95K

?

15

No

Large

67K

?

Apply Model

Class

Decision Tree

Deduction

10

Test Set Information Retrieval and Data Mining, SoSe 2015, S. Michel

19

Constructing Decision Tree • There are exponentially many decision trees for the training data. • Finding optimal tree is computationally infeasible. • Instead, use greedy algorithms: Series of local split operations to grow the tree. Not optimal, but there are efficient algorithms that create sufficiently accurate trees.

Information Retrieval and Data Mining, SoSe 2015, S. Michel

20

© Tan,Steinbach, Kumar

General Structure of Hunt’s Algorithm • Let Dt be the set of training records that reach a node t • General Procedure: – If Dt contains records that belong the same class yt, then t is a leaf node labeled as yt – If Dt is an empty set, then t is a leaf node labeled by the default class, yd – If Dt contains records that belong to more than one class, use an attribute test to split the data into smaller subsets. Recursively apply the procedure to each subset.

Tid Refund Marital Status

Taxable Income Cheat

1

Yes

Single

125K

No

2

No

Married

100K

No

3

No

Single

70K

No

4

Yes

Married

120K

No

5

No

Divorced 95K

Yes

6

No

Married

No

7

Yes

Divorced 220K

No

8

No

Single

85K

Yes

9

No

Married

75K

No

10

No

Single

90K

Yes

60K

10

Information Retrieval and Data Mining, SoSe 2015, S. Michel

Dt

?

21

Hunt’s Algorithm Don’t Cheat

Start with most frequent class as default class.

Tid Refund Marital Status

Taxable Income Cheat

1

Yes

Single

125K

No

2

No

Married

100K

No

3

No

Single

70K

No

4

Yes

Married

120K

No

5

No

Divorced 95K

Yes

6

No

Married

No

7

Yes

Divorced 220K

No

8

No

Single

85K

Yes

9

No

Married

75K

No

10

No

Single

90K

Yes

60K

© Tan,Steinbach, Kumar

10

Information Retrieval and Data Mining, SoSe 2015, S. Michel

22

Hunt’s Algorithm (2) Refund

Don’t Cheat

Yes Don’t Cheat

No Don’t Cheat

Tid Refund Marital Status

Taxable Income Cheat

1

Yes

Single

125K

No

2

No

Married

100K

No

3

No

Single

70K

No

4

Yes

Married

120K

No

5

No

Divorced 95K

Yes

6

No

Married

No

7

Yes

Divorced 220K

No

8

No

Single

85K

Yes

9

No

Married

75K

No

10

No

Single

90K

Yes

60K

© Tan,Steinbach, Kumar

10

Information Retrieval and Data Mining, SoSe 2015, S. Michel

23

Hunt’s Algorithm (3) Refund

Don’t Cheat

Yes

No Don’t Cheat

Don’t Cheat

Refund

Yes

No

© Tan,Steinbach, Kumar

Don’t Cheat Single, Divorced Cheat

Marital Status

Tid Refund Marital Status

Taxable Income Cheat

1

Yes

Single

125K

No

2

No

Married

100K

No

3

No

Single

70K

No

4

Yes

Married

120K

No

5

No

Divorced 95K

Yes

6

No

Married

No

7

Yes

Divorced 220K

No

8

No

Single

85K

Yes

9

No

Married

75K

No

10

No

Single

90K

Yes

60K

10

Married

Don’t Cheat

Information Retrieval and Data Mining, SoSe 2015, S. Michel

24

Hunt’s Algorithm (4) Refund

Don’t Cheat

Yes

No Don’t Cheat

Don’t Cheat

Refund

Refund

Yes Don’t Cheat Single, Divorced © Tan,Steinbach, Kumar

Yes

No

Cheat

Marital Status

Married

Don’t Cheat Single, Divorced

Don’t Cheat

No Marital Status

Don’t Cheat

Taxable Income Cheat

1

Yes

Single

125K

No

2

No

Married

100K

No

3

No

Single

70K

No

4

Yes

Married

120K

No

5

No

Divorced 95K

Yes

6

No

Married

No

7

Yes

Divorced 220K

No

8

No

Single

85K

Yes

9

No

Married

75K

No

10

No

Single

90K

Yes

60K

10

Married Don’t Cheat

Taxable Income

< 80K

Tid Refund Marital Status

>= 80K Cheat

Information Retrieval and Data Mining, SoSe 2015, S. Michel

25

Tree Induction • Greedy strategy. – Split the records based on an attribute test that optimizes certain criterion.

• Issues

© Tan,Steinbach, Kumar

– Determine how to split the records • How to specify the attribute test condition? • How to determine the best split?

– Determine when to stop splitting Information Retrieval and Data Mining, SoSe 2015, S. Michel

26

How to Specify Test Condition? • Depends on attribute types – Nominal – Ordinal – Continuous

© Tan,Steinbach, Kumar

• Depends on number of ways to split – 2-way split – Multi-way split

Information Retrieval and Data Mining, SoSe 2015, S. Michel

27

Splitting Based on Nominal Attributes • Multi-way split: Use as many partitions as distinct values. CarType Family

Luxury

© Tan,Steinbach, Kumar

Sports

• Binary split: Divides values into two subsets. Need to find optimal partitioning. {Sports, Luxury}

CarType {Family}

OR

{Family, Luxury}

Information Retrieval and Data Mining, SoSe 2015, S. Michel

CarType {Sports}

28

Splitting Based on Continuous Attributes • Different ways of handling continuous attributes – Discretization to form an ordinal categorical attribute • Static – discretize once at the beginning • Dynamic – ranges can be found by equal interval bucketing, equal frequency bucketing (percentiles), or clustering.

© Tan,Steinbach, Kumar

– Binary Decision: (A < v) or (A  v) • consider all possible splits and finds the best cut • can be more compute intensive

Information Retrieval and Data Mining, SoSe 2015, S. Michel

30

Splitting Based on Continuous Attributes Taxable Income > 80K?

Taxable Income? < 10K

Yes

> 80K

No

© Tan,Steinbach, Kumar

[10K,25K)

(i) Binary split

[25K,50K)

[50K,80K)

(ii) Multi-way split

Information Retrieval and Data Mining, SoSe 2015, S. Michel

31

Tree Induction • Greedy strategy. – Split the records based on an attribute test that optimizes certain criterion.

• Issues

© Tan,Steinbach, Kumar

– Determine how to split the records • How to specify the attribute test condition? • How to determine the best split?

– Determine when to stop splitting Information Retrieval and Data Mining, SoSe 2015, S. Michel

32

How to determine the Best Split Before Splitting: 10 records of class 0 10 records of class 1 Own Car? Yes

Car Type? No

Family

Student ID? Luxury

c1

Sports

© Tan,Steinbach, Kumar

C0: 6 C1: 4

C0: 4 C1: 6

C0: 1 C1: 3

C0: 8 C1: 0

C0: 1 C1: 7

C0: 1 C1: 0

...

c10 C0: 1 C1: 0

c11 C0: 0 C1: 1

c20

...

C0: 0 C1: 1

Which test condition is the best? Information Retrieval and Data Mining, SoSe 2015, S. Michel

33

How to determine the Best Split • Greedy approach: – Nodes with homogeneous class distribution are preferred

• Need a measure of node impurity:

© Tan,Steinbach, Kumar

C0: 5 C1: 5

C0: 9 C1: 1

Non-homogeneous,

Homogeneous,

High degree of impurity

Low degree of impurity

Information Retrieval and Data Mining, SoSe 2015, S. Michel

34

Selecting the Best Split • Let p(i | t) be the fraction of records belonging to class i at node t • Best split is selected based on the degree of impurity of the child nodes – p(0 | t) = 0 and p(1 | t) = 1 has high purity – p(0 | t) = 1/2 and p(1 | t) = 1/2 has the smallest purity (highest impurity)

• Intuition: high purity ⇒ small value of impurity measures ⇒ better split 35

© Tan,Steinbach, Kumar

Example of Purity

high impurity

high purity

Information Retrieval and Data Mining, SoSe 2015, S. Michel

36

Impurity Measures

Information Retrieval and Data Mining, SoSe 2015, S. Michel

37

© Tan,Steinbach, Kumar

Examples for Computing Entropy

C1 C2

0 6

P(C1) = 0/6 = 0

C1 C2

1 5

P(C1) = 1/6

C1 C2

2 4

P(C1) = 2/6

P(C2) = 6/6 = 1

Entropy = – 0 log 0 – 1 log 1 = – 0 – 0 = 0

P(C2) = 5/6

Entropy = – (1/6) log2 (1/6) – (5/6) log2 (1/6) = 0.65 P(C2) = 4/6

Entropy = – (2/6) log2 (2/6) – (4/6) log2 (4/6) = 0.92 Information Retrieval and Data Mining, SoSe 2015, S. Michel

38

© Tan,Steinbach, Kumar

Examples for computing GINI C1 C2

0 6

P(C1) = 0/6 = 0

C1 C2

1 5

P(C1) = 1/6

C1 C2

2 4

P(C1) = 2/6

P(C2) = 6/6 = 1

Gini = 1 – P(C1)2 – P(C2)2 = 1 – 0 – 1 = 0

P(C2) = 5/6

Gini = 1 – (1/6)2 – (5/6)2 = 0.278 P(C2) = 4/6

Gini = 1 – (2/6)2 – (4/6)2 = 0.444 Information Retrieval and Data Mining, SoSe 2015, S. Michel

39

Comparing Conditions • The quality of the split: the change in the impurity – Called the gain of the test condition

• • • • •

I( ) is the impurity measure k is the number of attribute values p is the parent node, vj is the child node N is the total number of records at the parent node N(vj) is the number of records associated with the child node

• Maximizing the gain ⇔ minimizing the weighted average impurity measure of child nodes • If I() = Entropy(), then Δ = Δinfo is called information gain Information Retrieval and Data Mining, SoSe 2015, S. Michel

40

How to Find the Best Split Before Splitting:

C0 C1

N00 N01

M0

A?

B?

Yes

No

Node N1

© Tan,Steinbach, Kumar

C0 C1

Node N2

N10 N11

C0 C1

N20 N21

M2

M1

Yes

No

Node N3 C0 C1

Node N4

N30 N31

C0 C1

M3

M12

N40 N41

M4 M34

Gain = M0 – M12 vs M0 – M34 Information Retrieval and Data Mining, SoSe 2015, S. Michel

41

Problems of maximizing Δ

Higher purity

Information Retrieval and Data Mining, SoSe 2015, S. Michel

42

Problems of Maximizing Δ • Impurity measures favor attributes with large number of values • A test condition with large number of outcomes might not be desirable – Number of records in each partition is too small to make predictions

• Solution 1: gain ratio = Δinfo / SplitInfo

P(vi) = the fraction of records at child; k = total number of splits

• Solution 2: restrict the splits to binary 43

Tree Induction • Greedy strategy. – Split the records based on an attribute test that optimizes certain criterion.

• Issues

© Tan,Steinbach, Kumar

– Determine how to split the records • How to specify the attribute test condition? • How to determine the best split?

– Determine when to stop splitting Information Retrieval and Data Mining, SoSe 2015, S. Michel

44

Stopping Criteria for Tree Induction • Stop expanding a node when all the records belong to the same class

• Stop expanding a node when all the records have the same or similar attribute values. In this case the class with “majority” wins.

Information Retrieval and Data Mining, SoSe 2015, S. Michel

45

Overfitting and Tree Pruning • Common problem with decision trees is that tree might be too tightly tailored to training data (and thus possibly to noise in data). – Good: error on training data might be very low – But what about test data previously unseen?

• Idea: Avoid tree becoming too fine-grained. • Solution 1: Stop splitting nodes early (i.e., preprocessing) • Solution 2: Build tree regularly and then prune parts of it (i.e., postprocessing) Information Retrieval and Data Mining, SoSe 2015, S. Michel

46

Example: Training Data

© Tan,Steinbach, Kumar

Example of overfitting due to noisy training data …..

*) wrong class Information Retrieval and Data Mining, SoSe 2015, S. Michel

47

© Tan,Steinbach, Kumar

Example: Two Different Decision Trees

Information Retrieval and Data Mining, SoSe 2015, S. Michel

48

Example: Test Data 1

1 2 1

Let’s see how the trees M1 and M2 perform on test and training data. M1: 0% error on training data, but 30% error on test data! Errors marked with 1 M2: 20% error on training data, but 10% error on test data! 2 table source: Tan,Steinbach, Kumar

Information Retrieval and Data Mining, SoSe 2015, S. Michel

49

2. (Naive) Bayes Classifier • A probabilistic framework for solving classification problems • Conditional Probability: P(C | A)  P( A, C )

© Tan,Steinbach, Kumar

P ( A) P ( A, C ) P( A | C )  P (C )

• Bayes theorem:

P( A | C ) P(C ) P(C | A)  P( A) Information Retrieval and Data Mining, SoSe 2015, S. Michel

50

Example of Bayes Theorem • Given: – A doctor knows that meningitis causes stiff neck 50% of the time – Prior probability of any patient having meningitis is 1/50,000 – Prior probability of any patient having stiff neck is 1/20

© Tan,Steinbach, Kumar

• If a patient has stiff neck, what’s the probability he/she has meningitis? P( S | M ) P( M ) 0.5 1 / 50000 P( M | S )    0.0002 P( S ) 1 / 20 Information Retrieval and Data Mining, SoSe 2015, S. Michel

51

Bayesian Classifiers • Consider each attribute and class label as random variables • Given a record with attributes (A1, A2,…,An)

© Tan,Steinbach, Kumar

– Goal is to predict class C – Specifically, we want to find the value of C that maximizes P(C| A1, A2,…,An )

• Can we estimate P(C| A1, A2,…,An ) directly from data? Information Retrieval and Data Mining, SoSe 2015, S. Michel

52

Bayesian Classifiers

• Approach: – compute the posterior probability P(C | A1, A2, …, An) for all values of C using the Bayes theorem P( A A  A | C ) P(C ) P(C | A A  A )  P( A A  A ) 1

1

2

2

n

n

1

2

n

© Tan,Steinbach, Kumar

– Choose value of C that maximizes P(C | A1, A2, …, An) – Equivalent to choosing value of C that maximizes P(A1, A2, …, An|C) P(C) • How to estimate P(A1, A2, …, An | C )? Information Retrieval and Data Mining, SoSe 2015, S. Michel

53

Naïve Bayes Classifier • Assume independence among attributes Ai when class is given: P(A1, A2, …, An |C) = P(A1| Cj) P(A2| Cj)… P(An| Cj)

© Tan,Steinbach, Kumar

– Can estimate P(Ai| Cj) for all Ai and Cj. – New point is classified to Cj if P(Cj)  P(Ai| Cj) is maximal. Information Retrieval and Data Mining, SoSe 2015, S. Michel

54

How to Estimate Probabilities from s al al u c c i i Data? ategor ategor ontinuo lass

• Class: P(C) = Nc/N

c

Tid

© Tan,Steinbach, Kumar

– e.g., P(No) = 7/10, P(Yes) = 3/10

Refund

c

c

c

Marital Status

Taxable Income

Evade

1

Yes

Single

125K

No

2

No

Married

100K

No

• For discrete attributes: P(Ai | Ck) = |Aik|/ Nc

3

No

Single

70K

No

4

Yes

Married

120K

No

5

No

Divorced

95K

Yes

6

No

Married

60K

No

– where |Aik| is number of instances having attribute Ai and belongs to class Ck – Examples:

7

Yes

Divorced

220K

No

8

No

Single

85K

Yes

9

No

Married

75K

No

10

No

Single

90K

Yes

k

10

P(Status=Married|No) = 4/7 P(Refund=Yes|Yes)=0 Information Retrieval and Data Mining, SoSe 2015, S. Michel

55

How to Estimate Probabilities from Data? • For continuous attributes: – Discretize the range into bins • one ordinal attribute per bin • violates independence assumption

k

– Two-way split: (A < v) or (A > v) • choose only one of the two splits as new attribute

© Tan,Steinbach, Kumar

– Probability density estimation: • Assume attribute follows a normal distribution • Use data to estimate parameters of distribution (e.g., mean and standard deviation) • Once probability distribution is known, can use it to estimate the conditional probability P(Ai|c) Information Retrieval and Data Mining, SoSe 2015, S. Michel

56

How to Estimate Probabilities from Data? t ca

• Normal distribution:

Tid

– One for each (Ai,ci) pair

© Tan,Steinbach, Kumar

• For (Income, Class=No): – If Class=No • sample mean = 110 • sample variance = 2975

Refund

o eg

a c i r

l

t ca

o eg

a c i r

l

co

in nt

u

s u o

Marital Status

Taxable Income

Evade

1

Yes

Single

125K

No

2

No

Married

100K

No

3

No

Single

70K

No

4

Yes

Married

120K

No

5

No

Divorced

95K

Yes

6

No

Married

60K

No

7

Yes

Divorced

220K

No

8

No

Single

85K

Yes

9

No

Married

75K

No

10

No

Single

90K

Yes

10

Information Retrieval and Data Mining, SoSe 2015, S. Michel

as l c

57

Example of Naïve Bayes Classifier Given a Test Record:

X  (Refund  No, Married, Income  120K)

naive Bayes Classifier:

© Tan,Steinbach, Kumar

P(Refund=Yes|No) = 3/7 P(Refund=No|No) = 4/7 P(Refund=Yes|Yes) = 0 P(Refund=No|Yes) = 1 P(Marital Status=Single|No) = 2/7 P(Marital Status=Divorced|No)=1/7 P(Marital Status=Married|No) = 4/7 P(Marital Status=Single|Yes) = 2/7 P(Marital Status=Divorced|Yes)=1/7 P(Marital Status=Married|Yes) = 0 For taxable income: If class=No: sample mean=110 sample variance=2975 If class=Yes: sample mean=90 sample variance=25

P(X|Class=No) = P(Refund=No|Class=No)  P(Married| Class=No)  P(Income=120K| Class=No) = 4/7  4/7  0.0072 = 0.0024

P(X|Class=Yes) = P(Refund=No| Class=Yes)  P(Married| Class=Yes)  P(Income=120K| Class=Yes) = 1  0  1.2  10-9 = 0

Since P(X|No)P(No) > P(X|Yes)P(Yes) Therefore P(No|X) > P(Yes|X)

=> Class = No

Information Retrieval and Data Mining, SoSe 2015, S. Michel

58

3. Support Vector Machines

© Tan,Steinbach, Kumar

Idea: Find a linear hyperplane (decision boundary) that will separate the data

Information Retrieval and Data Mining, SoSe 2015, S. Michel

59

Support Vector Machines One Possible Solution

© Tan,Steinbach, Kumar

B1

Information Retrieval and Data Mining, SoSe 2015, S. Michel

60

Support Vector Machines Another possible solution

© Tan,Steinbach, Kumar

B2

Information Retrieval and Data Mining, SoSe 2015, S. Michel

61

Support Vector Machines Other possible solutions

© Tan,Steinbach, Kumar

B2

Information Retrieval and Data Mining, SoSe 2015, S. Michel

62

Support Vector Machines B1

© Tan,Steinbach, Kumar

B2

• Which one is better? B1 or B2? • How do you define better? Information Retrieval and Data Mining, SoSe 2015, S. Michel

63

Support Vector Machines B1

B2

© Tan,Steinbach, Kumar

b21 b22

margin

b11

b12

• Find hyperplane maximizes the margin => B1 is better than B2 Information Retrieval and Data Mining, SoSe 2015, S. Michel

64

Support Vector Machines B1

  w x  b  0   w  x  b  1

  w  x  b  1

© Tan,Steinbach, Kumar

b11

  if w  x  b  1 1  f ( x)      1 if w  x  b  1  Information Retrieval and Data Mining, SoSe 2015, S. Michel

b12

Margin 

2  || w ||2 65

Summary Data Mining • Frequent Itemset and Association Rule Mining: – Apriori Principle and Algorithm

• Clustering: – K-means – Hierarchical clustering – DBSCAN (density based clustering)

• Classification: – Decision trees – Naïve Bayes Classifier – Support Vector Machines (SVMs) Information Retrieval and Data Mining, SoSe 2015, S. Michel

66