CS 277, Data Mining Exploratory Data Analysis
Padhraic Smyth Department of Computer Science Bren School of Information and Computer Sciences University of California, Irvine
2
Outline Assignment 1: Questions? Today’s Lecture: Exploratory Data Analysis – Analyzing single variables – Analyzing pairs of variables – Higher-dimensional visualization techniques
Next Lecture: Clustering and Dimension Reduction – Dimension reduction methods – Clustering methods
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
3
What are the main Data Mining Techniques? •
Descriptive Methods – – – – –
Exploratory Data Analysis, Visualization Dimension reduction (principal components, factor models, topic models) Clustering Pattern and Anomaly Detection ….and more
• Predictive Modeling – – – – –
Classification Ranking Regression Matrix completion (recommender systems) …and more
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
4
The Typical Data Mining Process
Problem Definition Defining the Goal
Understanding the Problem Domain
Data Definition Defining and Understanding Features
Creating Training and Test Data
Data Exploration Exploratory Data Analysis
Data Mining Running Data Mining Algorithms
Evaluating Results/Models
Model Deployment System Implementation And Testing
Evaluation “in the field”
Model in Operations Model Monitoring
Model Updating
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
5
Exploratory Data Analysis: Single Variables
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
6
Summary Statistics Mean: “center of data” Mode: location of highest data density Variance: “spread of data” Skew: indication of non-symmetry
Range: max - min Median: 50% of values below, 50% above Quantiles: e.g., values such that 25%, 50%, 75% are smaller
Note that some of these statistics can be misleading E.g., mean for data with 2 clusters may be in a region with zero data
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
7
Histogram of Unimodal Data 1000 data points simulated from a Normal distribution, mean 10, variance 1, 30 bins
1200
1000
800
600
400
200
0
6
7
8
9
10
11
12
13
14
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
8
Histograms: Unimodal Data 100 data points from a Normal, mean 10, variance 1, with 5, 10, 30 bins 25
40 35
20 30 25
15
20
10
15 10
5 5 0
6
7
8
9
10
11
12
13
0
6
7
8
9
10
11
12
13
12
10
8
6
4
2
0
6
7
8
9
10
11
12
13
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
9
Histogram of Multimodal Data 15000 data points simulated from a mixture of 3 Normal distributions, 300 bins
400 350 300 250 200 150 100 50 0
5
6
7
8
9
10
11
12
13
14
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
10
Histogram of Multimodal Data 15000 data points simulated from a mixture of 3 Normal distributions, 300 bins 6000
3500
3000
5000
2500 4000
2000 3000
1500 2000
1000 1000
0
500
5
6
7
8
9
10
11
12
13
14
400
0
5
7
8
9
10
11
12
13
14
11
12
13
14
160
350
140
300
120
250
100
200
80
150
60
100
40
50
20
0
6
5
6
7
8
9
10
11
12
13
14
0
5
6
7
8
9
10
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
11
Skewed Data 5000 data points simulated from an exponential distribution, 100 bins 450 400 350 300 250 200 150 100 50 0
0
1
2
3
4
5
6
7
8
9
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
12
Another Skewed Data Set 10000 data points simulated from a mixture of 2 exponentials, 100 bins 5000 4500 4000 3500 3000 2500 2000 1500 1000 500 0
0
20
40
60
80
100
120
140
160
180
200
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
13
Same Skewed Data after taking Logs (base 10) 10000 data points simulated from a mixture of 2 exponentials, 100 bins 350
300
250
200
150
100
50
0 -4
-3
-2
-1
0
1
2
3
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
14
What will the mean or median tell us about this data?
900 800 700 600 500 400 300 200 100 0
9
10
11
12
13
14
15
16
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
15
Issues with Histograms •
For small data sets, histograms can be misleading. Small changes in the data or to the bucket boundaries can result in very different histograms.
•
For large data sets, histograms can be quite effective at illustrating general properties of the distribution.
•
Can smooth histogram using a variety of techniques – E.g., kernel density estimation, which avoids bins – but requires some notion of “scale”
•
Histograms effectively only work with 1 variable at a time – Difficult to extend to 2 dimensions, not possible for >2 – So histograms tell us nothing about the relationships among variables
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
16
US Zipcode Data: Population by Zipcode 900
8000
K = 50
7000
K = 500
800 700
6000
600
5000 500
4000 400
3000 300
2000
200
1000 0
100
0
2
4
6
8
10
0
12
0
2
4
6
8
10
12 4
4
x 10
x 10
400
K = 50
350 300 250 200 150 100 50 0
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
17
Histogram with Outliers
Pima Indians Diabetes Data, From UC Irvine Machine Learning Repository
Number of Individuals
X values Padhraic Smyth, UC Irvine: CS 277, Winter 2014
18
Histogram with Outliers
Number of Individuals
Pima Indians Diabetes Data, From UC Irvine Machine Learning Repository
blood pressure = 0 ?
Diastolic Blood Pressure Padhraic Smyth, UC Irvine: CS 277, Winter 2014
19
Box Plots: Pima Indians Diabetes Data Two side-by-side box-plots of individuals from the Pima Indians Diabetes Data Set
Body Mass Index
Healthy Individuals
Diabetic Individuals Padhraic Smyth, UC Irvine: CS 277, Winter 2014
20
Box Plots: Pima Indians Diabetes Data Two side-by-side box-plots of individuals from the Pima Indians Diabetes Data Set
Body Mass Index
Plots all data points outside “whiskers” Upper Whisker
1.5 x Q3-Q1 Q3
Q2 (median)
Box = middle 50% of data
Q1 Lower Whisker
Healthy Individuals
Diabetic Individuals Padhraic Smyth, UC Irvine: CS 277, Winter 2014
21
Box Plots: Pima Indians Diabetes Data
healthy
Diastolic Blood Pressure
24-hour Serum Insulin
Plasma Glucose Concentration
Body Mass Index
diabetic
healthy
diabetic Padhraic Smyth, UC Irvine: CS 277, Winter 2014
22
Exploratory Data Analysis
Analyzing more than 1 variable at a time…
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
23
Relationships between Pairs of Variables • Say we have a variable Y we want to predict and many variables X that we could use to predict Y • In exploratory data analysis we may be interested in quickly finding out if a particular X variable is potentially useful at predicting Y • Options? – Linear correlation – Scatter plot: plot Y values versus X values
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
24
Linear Dependence between Pairs of Variables • Covariance and correlation measure linear dependence • Assume we have two variables or attributes X and Y and n objects taking on values x(1), …, x(n) and y(1), …, y(n). The sample covariance of X and Y is: 1 n Cov( X , Y ) ( x(i ) x )( y (i ) y ) n i 1
• The covariance is a measure of how X and Y vary together. – it will be large and positive if large values of X are associated with large values of Y and small X small Y
• (Linear) Correlation = scaled covariance, varies between -1 and 1 n
( x(i) x )( y(i) y )
( X ,Y )
i 1
( x(i ) x ) 2 ( y (i ) y ) 2 i 1 i 1 n
n
1 2
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
25
Correlation coefficient • Covariance depends on ranges of X and Y • Standardize by dividing by standard deviation • Linear correlation coefficient is defined as:
n
( x(i) x )( y(i) y )
( X ,Y )
i 1
( x(i ) x ) 2 ( y (i ) y ) 2 i 1 i 1 n
n
1 2
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
26
Data Set on Housing Prices in Boston (widely used data set in research on regression models)
1
CRIM
per capita crime rate by town
2
ZN
proportion of residential land zoned for lots over 25,000 ft2
3
INDUS
proportion of non-retail business acres per town
4
NOX
Nitrogen oxide concentration (parts per 10 million)
5
RM
average number of rooms per dwelling
6
AGE
proportion of owner-occupied units built prior to 1940
7
DIS
weighted distances to five Boston employment centres
8
RAD
index of accessibility to radial highways
9
TAX
full-value property-tax rate per $10,000
10
PTRATIO
pupil-teacher ratio by town
11
MEDV
Median value of owner-occupied homes in $1000's
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
27
Matrix of Pairwise Linear Correlations -1 0 +1 Crime Rate
Industry Nitrous oxide Average # rooms Proportion of old houses
Highway accessibility Property tax rate Student-teacher ratio
Data on characteristics of Boston housing
Percentage of large residential lots
Distance to employment centers
Median house value
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
28
Dangers of searching for correlations in high-dimensional data Simulated 50 random Gaussian/normal data vectors, each with 100 variables Results in a 50 x 100 data matrix Below is a histogram of the 100 choose 2 pairs of correlation coefficients Even if data are entirely random (no dependence) there is a very high probability some variables will appear dependent just by chance.
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
29
Examples of X-Y plots and linear correlation values
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
30
Examples of X-Y plots and linear correlation values
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
31
Examples of X-Y plots and linear correlation values Non-Linear Dependence
Lack of linear correlation does not imply lack of dependence
Linear Dependence
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
32
DATA SET 2 15
10
10
Y values
Y values
DATA SET 1 15
5
0
0
5
10
15
5
0
20
0
5
X values
10
10
5
5
10
20
15
20
DATA SET 4 15
Y values
Y values
DATA SET 3
0
15
X values
15
0
10
15
20
X values
5
0
0
5
10
X values
Anscombe, Francis (1973), Graphs in Statistical Analysis, The American Statistician, pp. 195-199. Padhraic Smyth, UC Irvine: CS 277, Winter 2014
33
Guess the Linear Correlation Values for each Data Set DATA SET 2 15
10
10
Y values
Y values
DATA SET 1 15
5
0
0
5
10
15
5
0
20
0
5
X values
10
10
5
5
10
20
15
20
DATA SET 4 15
Y values
Y values
DATA SET 3
0
15
X values
15
0
10
15
20
X values
5
0
0
5
10
X values
Anscombe, Francis (1973), Graphs in Statistical Analysis, The American Statistician, pp. 195-199. Padhraic Smyth, UC Irvine: CS 277, Winter 2014
34
Actual Correlation Values DATA SET 1
DATA SET 2
15
15
Correlation = 0.82
10
Y values
Y values
Correlation = 0.82
5
0
0
5
10
15
10
5
0
20
0
5
X values DATA SET 3
20
15
20
DATA SET 4 15
Correlation = 0.82
Correlation = 0.82
10
Y values
Y values
15
X values
15
5
0
10
0
5
10
15
20
X values
10
5
0
0
5
10
X values
Anscombe, Francis (1973), Graphs in Statistical Analysis, The American Statistician, pp. 195-199. Padhraic Smyth, UC Irvine: CS 277, Winter 2014
35
Summary Statistics for each Data Set Summary Statistics of Data Set 1 N = 11 Mean of X = 9.0 Mean of Y = 7.5 Intercept = 3 Slope = 0.5 Correlation = 0.82
Summary Statistics of Data Set 2 N = 11 Mean of X = 9.0 Mean of Y = 7.5 Intercept = 3 Slope = 0.5 Correlation = 0.82
Summary Statistics of Data Set 3 N = 11 Mean of X = 9.0 Mean of Y = 7.5 Intercept = 3 Slope = 0.5 Correlation = 0.82
Summary Statistics of Data Set 4 N = 11 Mean of X = 9.0 Mean of Y = 7.5 Intercept = 3 Slope = 0.5 Correlation = 0.82
Anscombe, Francis (1973), Graphs in Statistical Analysis, The American Statistician, pp. 195-199. Padhraic Smyth, UC Irvine: CS 277, Winter 2014
36
Conclusions so far? • Summary statistics are useful…..up to a point
• Linear correlation measures can be misleading
• There really is no substitute for plotting/visualizing the data
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
37
Scatter Plots • Plot the value of one variable against the other • Simple…but can be very informative, can reveal more than summary statistics • For example, we can… – See if variables are dependent on each other (beyond linear dependence) – Detect if outliers are present – Can color-code to overlay group information (e.g., color points by class label for classification problems)
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
38 5
2.5
(from US Zip code data: each point = 1 Zip code)
x 10
MEDIAN HOUSEHOLD INCOME
units = dollars
2
1.5
1
0.5
0
0
2
4
6
8
10
MEDIAN PERCAPITA INCOME
12
14 4
x 10
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
39
Constant Variance versus Changing Variance
variation in Y does not depend on X
variation in Y changes with the value of X e.g., Y = annual tax paid, X = income
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
40
Problems with Scatter Plots of Large Data appears: later apps older; reality: downward slope (more apps, more variance)
96,000 bank loan applicants
scatter plot degrades into black smudge ... Padhraic Smyth, UC Irvine: CS 277, Winter 2014
41
Problems with Scatter Plots of Large Data appears: later apps older; reality: downward slope (more apps, more variance)
96,000 bank loan applicants
scatter plot degrades into black smudge ... Padhraic Smyth, UC Irvine: CS 277, Winter 2014
42
Contour Plots (based on local density) can help recall: (same 96,000 bank loan apps as before)
shows variance(y) with x is indeed due to horizontal skew in density
unimodal
skewed
skewed
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
43
Scatter-Plot Matrices Pima Indians Diabetes data
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
44
Another Scatter-Plot Matrix
For interactive visualization the concept of “linked plots” is generally useful, i.e., clicking on 1 or more points in 1 window and having these same points highlighted in other windows
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
45
Using Color to Show Group Information in Scatter Plots Iris classification data set, 3 classes
Figure from www.originlab.com Padhraic Smyth, UC Irvine: CS 277, Winter 2014
46
Another Example with Grouping by Color
Figure from hci.stanford.edu Padhraic Smyth, UC Irvine: CS 277, Winter 2014
47
Outlier Detection • Definition of an outlier? No precise definition Generally….”A data point that is significantly different to the rest of the data” But how do we define “significantly different”? (many answers to this…..) Typically assumed to mean that the point was measured in error, or is not a true measurement in some sense
Outliers in 1 dimension
Outlier in 2 dimensions 9
8
7
Y VALUES
– – – –
6
5
4
3
2
1
2
3
4
5 X VALUES
6
7
8
9
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
48 0
0
5
10
15
0
20
0
5
10
Example 1: The Effect of Outliers on Regression X values
DATA SET 3
15
5
10
10
0
0
5
5
10
15
DATA SET 1
5 20
10
1
5
1
0
Y values
15
Y values
15
Y values
5
10
Y values
Y values
Y values
DATA SET 4
DATA SET 2
10
0
5
10
X values 0
0
5
10
15
0
20
0
15
2
X values 5
0
10
0
X values
X values
Least SquaresDATA Fit SET with the Outlier 3
DATA SET 4
15
5
20
10
15
20
X values
Least SquaresDATA Fit without the Outlier SET 3 15
1
10
10
10
1
5
0
0
5
10
X values
15
20
5
0
0
5
10
X values
Y values
15
Y values
15
Y values
Y values
2
15
DATA SET 1 15
15
X values
5
0
0
15
5
20
10
15
20
X values
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
49
Example 2: Least Square Fitting is Sensitive to Outliers 18 16
16 2 cost for this one datum
14 12
Heavy penalty for large errors
10 5 4 3 2 1 0 -20
8 6
4
-15
-10
-5
0
5
2 0 0
2
4
6
8
10
12
14
16
18
20
Slide courtesy of Alex Ihler Padhraic Smyth, UC Irvine: CS 277, Winter 2014
50
More Robust Cost Functions for Training Regression Models
(MSE)
(MAE)
Something else entirely, e.g., (Blue Line)
Slide courtesy of Alex Ihler Padhraic Smyth, UC Irvine: CS 277, Winter 2014
51
L1 is more Robust to Outliers than L2 18
L2, original data
16
L1, original data
14 12
L2, outlier data
10
L1, outlier data
8 6 4 2 0
0
2
4
6
8
10
12
14
16
18
20
Slide courtesy of Alex Ihler Padhraic Smyth, UC Irvine: CS 277, Winter 2014
52
Detection of Outliers in Multiple Dimensions 9
8
Y VALUES
7
6
5
4
3
2
1
2
3
4
5 X VALUES
6
7
8
9
• Detecting “multi-dimensional outliers” is generally difficult • In the example above, the blue point will not look like an outlier if we were to plot 1-dimensional histograms of Y or X – it only stands out in the 2d plot • Now consider the same situation but in 3 or more dimensions
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
53
Some Advice on Outliers • Use visualization (e.g., in 1d, 2d) to spot obvious outliers • Use domain knowledge and known constraints – E.g., Age in years should be between 0 and 120
• Use the model itself to help detect outliers – E.g., in regression, data points with errors much larger than the others may be outliers
• Use robust techniques that are not overly sensitive to outliers – E.g., median is more robust than the mean, L1 is more robust than L2, etc
• Automated outlier detection algorithms? …not always useful – E.g., fit probability density model to N-1 points and determine how likely the Nth point is – May not work well in high dimensions and/or if there are multiple outliers
• In general: for large data sets outliers you can probably assume that outliers are present and proceed with caution…. Padhraic Smyth, UC Irvine: CS 277, Winter 2014
54
Multivariate Visualization •
Multivariate -> multiple variables
•
2 variables: scatter plots, etc
•
3 variables: – – – –
•
4 variables: – –
•
3-dimensional plots Look impressive, but often not that useful Can be cognitively challenging to interpret Alternatives: overlay color-coding (e.g., categorical data) on 2d scatter plot
3d with color or time Can be effective in certain situations, but tricky
Higher dimensions – – – –
Generally difficult Scatter plots, icon plots, parallel coordinates: all have weaknesses Alternative: “map” data to lower dimensions, e.g., PCA or multidimensional scaling Main problem: high-dimensional structure may not be apparent in low-dimensional views
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
55
Using Icons to Encode Information, e.g., Star Plots Each star represents a single observation. Star plots are used to examine the relative values for a single data point The star plot consists of a sequence of equi-angular spokes, called radii, with each spoke representing one of the variables. Useful for small data sets with up to 10 or so variables 1 2 3 4
Price Mileage (MPG) 1978 Repair Record (1 = Worst, 5 = Best) 1977 Repair Record (1 = Worst, 5 = Best)
5 6 7 8
Headroom Rear Seat Room Trunk Space Weight
Limitations? Small data sets, small dimensions Ordering of variables may affect perception
9 Length
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
56
Another Example of Icon Plots
Figure from statsoft.com Padhraic Smyth, UC Irvine: CS 277, Winter 2014
57
Combining Scatter Plots and Icon Plots
Figure from statsoft.com Padhraic Smyth, UC Irvine: CS 277, Winter 2014
58
Chernoff Faces •
Variable values associated with facial characteristic parameters, e.g., head eccentricity, eye eccentricity, pupil size, eyebrow slant, nose size, mouth shape, eye spacing, eye size, mouth length and degree of mouth opening
• Limitations? – Only up to 10 or so dimensions – Overemphasizes certain variables because of our perceptual biases
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
59
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
60
Parallel Coordinates Method Epileptic Seizure Data
1 (of n) cases Interactive “brushing” is useful for seeing distinctions
(this case is a “brushed” one, with a darker line, to standout from the n-1 other cases)
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
61
More elaborate parallel coordinates example (from E. Wegman, 1999). 12,000 bank customers with 8 variables Additional “dependent” variable is profit (green for positive, red for negative)
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
62
Interactive “Grand Tour” Techniques • “Grand Tour” idea – – – – –
Cycle continuously through multiple projections of the data Cycles through all possible projections (depending on time constraints) Projects can be 1, 2, or 3d typically (often 2d) Can link with scatter plot matrices (see following example) Asimov (1985)
• Example on following 2 slides –
7 dimensional physics data, color-coded by group, shown with (a) Standard scatter matrix (b) 2 static snapshots of grand tour
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
63
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
64
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
65
Visualization of an email network using 2-dimensional graph drawing or “embedding”
Data from 500 researchers at Hewlett-Packard over approximately 1 year. Various structural elements of the network are apparent Padhraic Smyth, UC Irvine: CS 277, Winter 2014
66
Exploratory Data Analysis
Visualizing Time-Series Data
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
67
Time-Series Data: Example 1 Historical data on millions of miles flown by UK airline passengers …..note a number of different systematic effects
Summer “double peaks” (favor early or late)
Summer peaks
steady growth trend New Year bumps
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
68
Time-Series Data: Example 2 Data from study on weight measurements over time of children in Scotland
Experimental Study: More milk -> better health? 20,000 children: 5k raw, 5k pasteurize, 10k control (no supplement)
mean weight vs mean age for 10k control group
Weight
Would expect smooth weight growth plot. Plot shows an unexpected pattern (steps), not apparent from raw data table.
Age
Why do the children appear to grow in spurts?
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
69
Time-Series Data: Example 3 (Google Trends) Search Query = whiskey
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
70
Time-Series Data: Example 4 (Google Trends)
Search Query = NSA
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
71
Spatial Distribution of the Same Data (Google Trends) Search Query = whiskey
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
72
Summary on Exploration/Visualization • Always useful and worthwhile to visualize data – – – –
human visual system is excellent at pattern recognition gives us a general idea of how data is distributed, e.g., extreme skew detect “obvious outliers” and errors in the data gain a general understanding of low-dimensional properties
• Many different visualization techniques • Limitations – generally only useful up to 3 or 4 dimensions – massive data: only so many pixels on a screen - but subsampling is useful
Padhraic Smyth, UC Irvine: CS 277, Winter 2014
