168

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 15, NO. 1, FEBRUARY 2014

Automated Detection of Driver Fatigue Based on Entropy and Complexity Measures Chi Zhang, Hong Wang, and Rongrong Fu

Abstract—This paper presents a real-time method based on various entropy and complexity measures for detection and identification of driving fatigue from recorded electroencephalogram (EEG), electromyogram, and electrooculogram signals. The complexity features were used to distinguish whether the subjects are experienced drivers by calculating the Lempel–Ziv complexity of EEG approximate entropy (ApEn). Different threshold values can be set for the two kinds of drivers individually. The entropy-based features, namely, the wavelet entropy (WE), the peak-to-peak value of ApEn (PP-ApEn), and the peak-to-peak value of sample entropy (PP-SampEn), were extracted from the collected signals to estimate the driving fatigue stages. We proposed WE in a sliding window (WES), PP-ApEn in a sliding window (PP-ApEnS), and PP-SampEn in a sliding window (PP-SampEnS) for real-time analysis of driver fatigue. The real-time features obtained by WE, PP-ApEn, and PP-SampEn with sliding window were applied to artificial neural network for training and testing the system, which gives four situations for the fatigue level of the subjects, namely, normal state, mild fatigue, mood swing, and excessive fatigue. Then, the driver fatigue level can be determined in real time. The accuracy of estimation is about 96.5%–99.5%. Receiver operating characteristic (ROC) curve was used to present the performance of the neural network classifier. The area under the ROC curve is 0.9931. The results show that the developed method is valuable for the application of avoiding some traffic accidents caused by driver’s fatigue. Index Terms—Driver fatigue, electroencephalogram (EEG), electromyogram (EMG), electrooculogram (EOG), entropy, neural network.

I. I NTRODUCTION

F

ATIGUE is a feeling of extreme physical or mental tiredness. Almost everyone becomes fatigued at some time, but driver’s fatigue is a serious problem that leads to thousands of automobile crashes each year [1]–[3]. Fatigue process is often a change from the alertness and vigor state to the tiredness and weakness state. It is not only accompanied by drowsiness but also has a negative impact on mood. There have been studies to detect and quantify fatigue from the measurement of physiology variables such as electroencephalogram (EEG), electrooculogram (EOG), and electromyogram (EMG) [4]–[6]. However, simultaneous usage

Manuscript received December 31, 2012; revised April 8, 2013 and June 23, 2013; accepted July 18, 2013. Date of publication September 5, 2013; date of current version January 31, 2014. This work was supported in part by the National Science Foundation of China under Grant 61071057 and in part by the Innovation Groups of the Chinese Ministry of Education. The Associate Editor for this paper was R. I. Hammoud. (Corresponding author: H. Wang.) The authors are with the Department of Mechanical Engineering and Automation, Northeastern University, Shenyang 110819, China (e-mail: zhch_ [email protected]; [email protected]; [email protected]). Digital Object Identifier 10.1109/TITS.2013.2275192

and real-time analysis of EEG, EMG, and EOG signals were not given. This is the novelty that this paper brings. The EEG signals do a pretty good job of state discrimination. All the physical and mental activities associated with driving are reflected in EEG signals [4]. The EMG signals are influenced by muscle activities; a person gets lower tonus of EMG when his fatigue process gets further [7]. The EOG signals can be very useful to detect drowsiness. It has been observed that eye movement decreases while blink rate increases as a person enters into the state of fatigue [8]. Obviously, the simultaneous usage of EEG, EMG, and EOG signals can increase the accuracy of identification and classification results. The recorded physiological signals are nonlinear, timevarying, space-varying, and nonstationary in nature. Nonlinear dynamical analysis can provide complementary information about the dynamics under physiological or psychological states compared with classical linear time series analysis methods such as Fourier or spectral analysis [9], [10]. Nonlinear dynamical analysis techniques derived from the theory of nonlinear dynamical systems such as the correlation integral, Lyapunov exponents, and correlation dimension have been recently used in a number of fields of application. Assessment of driver’s fatigue is one of the special application areas [4], [11], [12]. One approach to the nonlinear estimation of dynamical EEG, EOG, and EMG activity is complexity analysis. Among complexity analysis approaches, entropy-based algorithms have been useful and robust estimators for evaluating regularity or predictability [13]. Shannon entropy (SE) is a disorder quantifier and is a measure of the flatness of energy spectrum in the wavelet domain [4], [14]. In this paper, we use the energy sequence distribution to replace the distribution of the probability distribution of SE to calculate wavelet entropy (WE). Approximate entropy (ApEn) and its refined version, i.e., sample entropy (SampEn), were developed as practically tractable physiological measures in view of their robustness to noise and finitude of data sets and applicability to stochastic, nonlinear deterministic, and composite processes [15], [16]. Lempel–Ziv complexity (LZC) is an approach to the problem that links the complexity of a specific sequence to the gradual buildup of new patterns along the given sequence [17]. It can show the approximation degree between finite sequence and random sequence. The greater the LZC of the sequence is, the closer to random sequence it will be. EEG LZC, which reflects the amount of the EEG information, can reveal brain activity regularity [18]. This paper presents a methodology for automatic detection of normal and fatigue states from recorded EEG, EOG, and EMG signals. The complexity features were used to distinguish whether the subjects are experienced drivers by calculating the

1524-9050 © 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

ZHANG et al.: AUTOMATED DETECTION OF DRIVER FATIGUE BASED ON ENTROPY AND COMPLEXITY MEASURES

169

LZC of EEG approximate entropy. Different threshold values can be set for the two kinds of drivers individually. Three entropy-based features, namely, the WE, the peak-to-peak value of ApEn (PP-ApEn), and the peak-to-peak value of SampEn (PP-SampEn), were extracted from the collected signals. We proposed WE in a sliding window (WES), PP-ApEn in a sliding window (PP-ApEnS), and PP-SampEn in a sliding window (PP-SampEnS) for real-time analysis of driver fatigue. The real-time calculation results based on WE, PP-ApEn, and PPSampEn with sliding window are fed to an artificial neural network (ANN) classifier, and then, the driver fatigue level can be determined in real time. Fig. 1. Experimental setting.

II. E XPERIMENTS AND DATA A. Experiment Design The experiments were conducted on 20 healthy male subjects with ages ranging from 20 to 35 years and a mean age of 27.5. Six subjects were asked to drive in a busy traffic of simulated driving situation to get different types of driving activities to evaluate every driver’s skill and performance. Our aim about the group is to distinguish whether the subjects are experienced drivers using the physiological signals. Eight subjects have been chosen to obtain test signals. Our aim about the group is to find fatigue features of the test signals to develop a new method that can estimate the fatigue state of human drivers in real time. The other six subjects have been chosen for validating the estimation method and obtaining better generalized performance of the system. All the subjects were asked to drive for 2 h to collect the EEG, EOG, and EMG signals in the simulated driving situation. During the experiments, the participants’ responses and the time they appeared were recorded for validation. When subjects appear bored, anxious, agitated and in a restless mood, or grimacing, it is considered that they are suffering mood swings. The time is recorded. It is informed by the subjects and is obtained by observation. When subjects began to feel fatigued, it is considered that they are in the mild fatigue state. The time is recorded. It is informed by the subjects. When subjects show the fatigue performances that humans cannot control such as yawn and doze, it is considered that they are in the excessive fatigue state. The time is recorded. It is obtained by observation. In the experiments, the EEG, EOG, and EMG were simultaneously recorded to a data acquisition device worn on the body and then sent to an external laptop by wireless Bluetooth for further processing. EEGs, EOGs, and EMGs were collected from electrodes that were placed on the occiput, upper eyelid, and neck zone, respectively (see Fig. 1). The sampling frequency was kept at 200 Hz with 16 bit analog-to-digital conversion. EEG electrodes were placed in accordance with the International 10–20 System: O1, O2. During driver fatigue, occipital and parietal alpha spread to more anterior regions such as the centrofrontal and temporal regions [8]. It may cause a significant change in entropy and complexity. This is the reason why we put the electrodes on O1, O2. EMG electrodes were placed at the back of the neck considering the subjects’ comfort, driving safety, and the quality of the test signals since

the limbs of the drivers often move irregularly and putting the electrodes into the clothes to contact torso is not suitable for long-term measurement (the subjects may feel uncomfortable, and there may be poor contact between the electrodes and the skin because of sweating and friction). B. Data Preprocessing Since the collected signals are disturbed by noise in the experiments, we use wavelet transform to achieve EEG signal filtering to make further analysis. Wavelet transform is particularly effective for representing various aspects of signals, such as trends, discontinuities, and repeated patterns, where other signal processing approaches fail or are not as effective. It is particularly powerful for nonstationary signal analysis [19]. The wavelet transform decomposes a signal into a set of basic functions called wavelets. These basic functions are obtained by dilations, contractions, and shifts of a unique function called wavelet prototype. The wavelet transform is divided into the continuous wavelet transform and the discrete wavelet transform (DWT). DWT analyzes the signal at different frequency bands with different resolutions by decomposing the signal into a coarse approximation and detailed information. DWT employs the sets of scaling functions and wavelet functions, which are associated with the low-pass filter g[n] and the high-pass filter h[n], respectively. After filtering, half of the samples will be eliminated according to Nyquist’ rule, and the coarse approximation and detailed information can be distinguished. The procedure forms one level of wavelet decomposition. The mathematical expressions are given by  x[n] · h[2k − n] (1) Yhigh [k] =  Ylow [k] = x[n] · g[2k − n] (2) where Yhigh [k] and Ylow [k] are the outputs of the high- and lowpass filters, respectively. In this paper, we used the Daubechies wavelet of order 6 (db6) because of its efficiency. The number of decomposition levels was chosen to be five (see Fig. 2). The EEG signals were decomposed into details D1–D5 and approximations A1–A5. A2 decomposition is within the EEG range we care about (0–30 Hz).

170

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 15, NO. 1, FEBRUARY 2014

B. PP-ApEnS ApEn, which was proposed by Pincus in 1991 [20], is a nonlinear dynamic parameter to measure data regularity. It assigns a nonnegative number to represent the complexity of a time series and reflect the probability of occurrence of new information. Studies have shown that ApEn can characterize changes in the physiological state of the human body. A robust estimate of ApEn can be obtained by using short noisy data sets. Let us consider a time series {x(n) = x(1), x(2), . . . , x(N )}. ApEn is given by the formula ApEn(m, r, N ) =

1 N −m+1

A. WES If we decompose a signal at m levels using wavelet transform, assume that the wavelet coefficient vector at the jth level is wj = (wj1 , wj2 ), where 1 represents approximation information, and 2 represents detail information. We use a sliding window with the length of sampling length for realtime analysis. The energy of the wavelet coefficient vector in the sliding window is defined as Ejk = wjk  =

l 

|wjk |2 (j = 1, 2, . . . , m k = 1, 2) (3)

n=1

where l is the length of the sliding window (l = sampling frequency). The energy sequence distribution in the sliding window is defined as the normalized energy of every wavelet coefficient vector pi = Ejk /E (i = 1, 2, . . . , 2m)

(4)

where E is the total energy, i.e., E=

m  2 

(8)

where d|X(i), X(j)| is a measure of the distance between X(i) and X(j). Define the distance as the maximum difference between corresponding elements. m is the embedding dimension, r is the scale or tolerance parameter, and N is the number of data points in phase space. In the calculation of ApEn, the parameters are typically chosen as m = 2 and r = 0.2 ∗ SD (SD is the standard deviation of the original time series x(n)). We also use a sliding window with the length of sampling length (1 s) to do real-time analysis for the ApEn of EEG, EOG, and EMG. However, the ApEn versus time curve is not so easy to see the trends [21]. Therefore, it requires the following step to calculate the PP-ApEn. In this paper, we define the PP-ApEn to be the difference between the maximum and minimum values in a moving window, i.e., PPApEn = max (ApEn(t)) − min (ApEn(t)) ,

Ejk .

(5)

Combined with the definition of entropy, replace the distribution of the probability distribution with the energy sequence distribution pi . The entropy based on the energy distribution is defined as WE, i.e., WE = −

(7)

X(i) = [x(i), x(i + 1), . . . , x(i + m − 1)] , X(i) ∈ Rm (9) X(j) = [x(j), x(j + 1), . . . , x(j + m − 1)] , X(j) ∈ Rm (10)

a≤t≤a+l

j=1 k=1

2m 

i=1 N −m 

where Bi = number of j such that d|X(i), X(j)| ≤ r. The two m-dimensional vectors are defined as follows:

III. F EATURE E XTRACTION

2

log Cim (r)

1 log Cim+1 (r) N − m i=1 Bi Cim (r) = N −m+1 −

Fig. 2. Subband decomposition of DWT implementation. g[n] is the low-pass filter; h[n] is the high-pass filter.

N −m+1 

pi log2 pi .

(6)

i=1

In this paper, we employ the sliding window with the length to be 200 points (1 s) to analyze experimental data. The window moves along the data points step by step to calculate the WE (m = 5) for EOG (j = 1, 2, 3, 4, 5), EMG (j = 1, 2, 3, 4, 5), and EEG (0-30 Hz j = 3, 4, 5) signals. Then we can obtain the entropy time-series of the experimental data.

a≤t≤a+l

∀ApEn(t) ∈ C[a, a + l]

(11)

where a represents the translation factor, l is the width of the sliding window, and C[a, b] is a set of ApEn(t) at [a, a + l]. C. PP-SampEnS SampEn, which was proposed by Richman and Moornan [16], is a new measure of system complexity, and it is designed to reduce the error of ApEn. It has a better consistency with the random part, which is known. SampEn is a similar measure with ApEn, but it has improved accuracy. SampEn is given by the formula SampEn(m, r, N ) = − ln

B m+1 (r) . B m (r)

(12)

ZHANG et al.: AUTOMATED DETECTION OF DRIVER FATIGUE BASED ON ENTROPY AND COMPLEXITY MEASURES

Define B m (r) as

171

IV. C LASSIFICATION D ECISION

B m (r) =

N −m  1 C m (r). N − m i=1 i

A. State Classification (13)

In the calculation of SampEn, the parameters are typically chosen as m = 2 and r = 0.2 ∗ SD. We still use a sliding window with the length of sampling length (1 s) to do real-time analysis for the SampEn of EEG, EOG, and EMG. However, the SampEn versus time curve is not so easy to see the trends [21]. Therefore, it requires the following step to calculate the PP-SampEn. In this paper, we define the PP-SampEn to be the difference between the maximum and minimum values in a moving window, i.e., PPSampEn = max (SampEn(t)) − min (SampEn(t)) , a≤t≤a+l

a≤t≤a+l

∀SampEn(t) ∈ C[a, a + l]

(14)

where a represents the translation factor, l is the width of the sliding window, and C[a, b] is a set of SampEn(t) at [a, a + l]. D. LZC The LZC algorithm is a measure proposed by Lempel and Ziv for estimating a complexity measure [17]. The LZC algorithm is described as follows [22]. 1) For a given binary sequence P = [x(1), x(2), . . . , x(n)], S and Q denote its two subsequences. Initially, S = x(1), and Q = x(2). Let SQ denote the concatenation of S and Q, whereas the sequence SQv is derived from SQ after its last character is deleted (v means the operation to delete the last character in the sequence). 2) Judge whether Q is a subsequence of SQv. If Q is a subsequence of SQv, it means that the characters of Q can be copied from S. Then cascade the next character of P to Q; If Q is not a subsequence of SQv, it means that the characters of SQv are inserted characters. Cascade Q to S, S = SQ. Reconstruct Q. Then Q is the next character of P . 3) Repeat (2) and (3) until the last character of P is included in Q. 4) Every time when Q is cascaded to S, a new pattern emerges. The number of new patterns is given by the measure of complexity c(n). In order to obtain the complexity measure, which is independent of the length of sequence, c(n) should be normalized. Lempel and Ziv have proved that c(n) tends to the fixed value of n/ logl n as n → ∞ (for a binary sequence, l = 2). The normalized formula is as follows: LZC =

c(n) logl n . n

(15)

In this paper, the EEG samples are converted into the binary sequence P . The complex calculation was used to distinguish whether the subjects are experienced drivers by calculating the complexity of EEG ApEn.

In this paper, we use WE to analyze the mood swings reflected by EEG signals, SampEn to analyze the fatigue time of EEG signals, ApEn to analyze the fatigue time of EMG signals, and WE to analyze the fatigue time of EOG signals. According to the different patterns of these entropies, we define four fatigue levels: normal state (level 1), mild fatigue (level 2), mood swing (level 3), and excessive fatigue (level 4). B. Neural Network Classifier The neural network classifier is a kind of a statistical machine learning method. It does not require statistical characteristics of samples and a priori knowledge of related fields. There is no need to determine the decision sequence. It has high recognition accuracy and fast recognition speed [23]. This is the reason why we use a neural network classifier to recognize the four fatigue levels. Artificial neural network (ANN) is a theoretical mathematical model of human brain and its activities. It is a nonalgorithmic, nonlinear, and adaptive system, which is constituted by appropriate interconnections of a large number of processing units. A unit is a neuron that can be expressed as A = f (WP + B)

(16)

where W represents the weight vector, B represents the threshold vector, P is the input vector, and f is the transfer function. Most neural networks have some sort of a “training” rule, whereby the weights of connections are adjusted so that the error between the desired output (the actual operating condition) and the network output (the estimated operating condition) for all sets of training data is minimized. The training is a process, which establishes the nonlinear mapping between the values of the performance parameters and the corresponding operating condition of the process valve [24]. The training process requires learning the samples to adjust the weight w and the threshold b. Learning in ANNs is accomplished through special training algorithms developed based on learning rules presumed to mimic the learning mechanisms of biological systems [25]. There are many different types and architectures of neural networks varying fundamentally in the way they learn, the details of which are well documented in the literature [26]–[29]. In this paper, neural network relevant to the application being considered (i.e., classification of driving state based on entropies) was employed for designing classifiers, i.e., the multilayer perceptron neural network (MLPNN). The backpropagation training algorithm was used to train the neural network classifier. Performance parameters were obtained for both the training and validation sets each time a new network topology was examined. Computer programs that we have written for the training algorithm based on backpropagation of error and Levenberg–Marquardt were used to develop the MLPNN. The curve of the mean square error (MSE) versus iteration is called the training curve. The training MSE curve of the neural network is shown in Fig. 8. As the network learns, the error converges to zero or a small constant. A neural network

172

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 15, NO. 1, FEBRUARY 2014

Fig. 3. Network architecture.

is subject to what is known as the statistical phenomenon of overfitting or overtraining. If a network overfits or memorizes the training data, its generalized performance (i.e., prospective prediction performance) on other sample populations is likely to be severely compromised. Therefore, the most important criterion is choosing the number of iterations for training. Cross validation is one of the most powerful methods to stop the training. In principle, the training curve exponentially decreases to zero or a small constant. Just how small in magnitude this constant is depends on the situation, and judgement must be used to find what error value is appropriate for the problem. When the error in the cross validation has increased, the training should be stopped because the point of best generalization has been reached [7]. The architecture of the MLPNN consists of one input layer, one or more hidden layers, and one output layer. When driver fatigue appears, there will be mutations in the extracted characteristics, i.e., the neural network inputs may not be continuous. Hence, four-layer architecture ANNs were adopted for the present study (one input, one output, and two hidden layers). WE values of EEG and EOG signals, ApEn values of EMG, and SampEn values of EEG are used as inputs. The output is the fatigue levels. Unlike the input and output layers, one starts with no prior knowledge as to the number of hidden layers. A network with too few hidden nodes would be incapable of differentiating between complex patterns leading to only a linear estimate of the actual trend. In contrast, if the network has too many hidden nodes, it will follow the noise in the data due to overparameterization, leading to poor generalization for untrained data. With increasing number of hidden layers, training becomes excessively time consuming. The most popular approach to finding the optimal number of hidden layers is by trial and error [27]. In this paper, we begin with a small number of hidden nodes and build on as needed to meet the model accuracy demand. Finally, we obtained the number of hidden nodes: S1 = 9 and S2 = 9. Fig. 3 shows the schematic of the network structure. The network used in the analysis had the following training parameters: adaptive learning coefficient: 0.08; momentum coefficient: 0.95; MSE: 0.001; activation function: tangent sigmoid.

Fig. 4.

WE versus time curve. (a) EEG WE. (b) EOG WE. (c) EMG WE.

C. ROC Curve The prediction accuracy in the test set is usually used to measure the predictive ability of the classifier. Additionally, in the driver fatigue detection application, we want the classifier to provide classification reliability, similarity, or the value estimation of each sample classification quality. ROC analysis is such a kind of reliability estimation. ROC curves regard each detection result as the possible critical value of diagnosis. True positive (i.e., sensitivity) is plotted on the Y -axis, and false positive (i.e., 1-specificity) is plotted on the X-axis. The area under the ROC curve (AUC) is accepted as a fair indicator to measure the classifier performance since it is invariant to the operating conditions such as different misclassification costs and skewed class distribution [30]. V. R ESULTS WE has a better effect on calculating the EOG fatigue time and reflecting mood swings (see Fig. 4). As shown in Fig. 4(a), if WE > 0.0015, it is considered that the subject is in the mood swing state. PP-SampEn has a better effect on calculating the EEG fatigue time (see Fig. 6). PP-ApEn has a better effect on calculating the EMG fatigue time (see Fig. 5). As shown in Fig. 4, we can clearly see a significant change in entropy after about 5000 s. At that time, subjects are in fatigue stage. According to the statistical regularity, the entropy values of the EEG signal decrease. The brain is less active, and the subject is from the alert state to the sleepy state. The emotion is clearly reflected in the EEG entropies. The entropy values of the EOG signal increase because blink frequency increases. After about 6000 s, there is a new fatigue stage, which we defined as excessive fatigue [see Fig. 4(a) and (b)]. At this stage, there are less sharp rises in Fig. 4(a) and more sharp rises in Fig. 4(b). The entropy values of the EMG signal are chaotic because of driving action’s complexity, but the tonic of muscle also decreases [Fig. 4(c)].

ZHANG et al.: AUTOMATED DETECTION OF DRIVER FATIGUE BASED ON ENTROPY AND COMPLEXITY MEASURES

173

TABLE I D RIVER FATIGUE T IME E STIMATED BY THE D IFFERENT E NTROPY-BASED M EASURES

TABLE II S UBJECT ’ S R ESPONSE R ECORD

Fig. 5. PP-ApEn versus time curve. (a) Peak-to-peak values of EEG ApEn. (b) Peak-to-peak values of EOG ApEn. (c) Peak-to-peak values of EMG ApEn.

Fig. 7. Complexity versus time curve. (a) Inexperienced driver complexity versus time curve. (b) Experienced driver complexity versus time curve.

Fig. 6. PP-SampEn versus time curve. (a) Peak-to-peak values of EEG SampEn. (b) Peak-to-peak values of EOG SampEn. (c) Peak-to-peak values of EMG SampEn.

As shown in Fig. 5, there is a significant change in the EEG and EOG PP-ApEn curve after about 4000 s. After about 3000 s, we can see that the PP-ApEn of EMG has been conspicuously changed. To estimate the EMG fatigue, PP-ApEn is better. As shown in Fig. 6, there is a significant change in the EEG and EOG PP-SampEn curve after about 4000 s. After about 3000 s, we can also see that the PP-SampEn of EMG has been changed. It is the driving fatigue time calculated by the SampEn method. To estimate the EEG fatigue, PP-SampEn is better. However, the EMG result is less conspicuous than PP-ApEn. Table I shows the driver fatigue time estimated by the integrated approach of different entropies. Table II shows the sub-

ject’s responses record in the experiment. Not coincidentally, the corresponding time of the different states in Table I can match with it. The trend of the experienced driver complexity curve is tantamount to the latter part [the bold part of the curve in Fig. 7(a)] of the inexperienced driver complexity curve, and this reflects the fact that experienced drivers do not need the prior adjusting stage. This result provides the basis for the distinction between the two kinds of drivers. The training MSE curve of the neural network is shown in Fig. 8. When the error in the cross validation does not decrease or even increases, the training should be stopped because the point of best generalization has been reached, or it will bring the overfitting of training data. The trained network was tested with EEG, EOG, and EMG signals. As a result, it was seen that, by observing the output vector produced by the network, several types of EEG, EOG, and EMG recordings have been tested to develop network. The responses of the network to these test signals are shown in Table III. Figs. 9–12 below are the signals identified by the neural network classifier. They are the normal state, the mild fatigue, the mood swing, and the excessive fatigue levels, respectively.

174

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 15, NO. 1, FEBRUARY 2014

Fig. 8. MSE versus iteration number. TABLE III T EST S IGNALS O BTAINED BY T RAINED ANN AND U SED EEG, EMG, AND EOG

Fig. 10. Mild fatigue signals obtained by classification. (a) EEG signal. (b) EOG signal. (c) EMG signal.

Fig. 11. Mood swing signals obtained by classification. (a) EEG signal. (b) EOG signal. (c) EMG signal.

Fig. 9. Normal signals obtained by classification. (a) EEG signal. (b) EOG signal. (c) EMG signal.

As shown in Table III, the subjects are corresponding to different responses due to different initial conditions and individual differences. According to these results, the accuracy of the network classification, which was calculated as a mean value ± standard deviation (SD), is presented in Table IV.

As shown in Table IV, the classification accuracy of ANN on the test data is about 96.5%–99.5%. This final table clearly shows that, when EEG signals are used with EOG and EMG signals as the bases of input parameters of ANN, the accuracy percentage is evidently increased with respect to the classification system using only EEG signals (95%–96% [31]). Fig. 13 shows the ROC curve for all the subjects, and the corresponding AUC is 0.9931. VI. D ISCUSSION As we know, the entropy values of EEG signal decrease when the driving fatigue appears. Fatigue has a negative impact on all

ZHANG et al.: AUTOMATED DETECTION OF DRIVER FATIGUE BASED ON ENTROPY AND COMPLEXITY MEASURES

Fig. 12. Excessive fatigue signals obtained by classification. (a) EEG signal. (b) EOG signal. (c) EMG signal. TABLE IV P ERFORMANCE OF THE ANN

Fig. 13. ROC curve.

areas of function, including mood and physical function. Thus, fatigue is usually accompanied by mood swings, which cause the entropy values of EEG signal to increase again. According to the statistical regularity, for different initial states of different people, the regular pattern of fatigue is the same, but there is the matter of different corresponding times. As shown in Fig. 6(a), to identify the peak before fatigue is easier, without retraining or relearning for different people.

175

After about 6000 s, there is a new fatigue stage, which we defined as excessive fatigue [see Fig. 4(a) and (b)]. At this stage, there are less sharp rises in Fig. 4(a) and more sharp rises in Fig. 4(b). It means that there are less mood swings and more blinks. The driver is in the drowsiness state. In the simulated driving experiment, the subject’s car often collided with other cars or in the guardrails. Excessive driving fatigue can easily lead to traffic accidents. Therefore, if we can identify it before the time drivers get into the fatigue process, it is an effective way to avoid traffic accidents by alarming or massage to help relieve fatigue. However, because of the accuracy of estimation (cannot be 100%), the difference between the fatigue time of estimation and subjective assessment, and the period of time from signal acquisition to fatigue identification, we cannot guarantee that identification of driver fatigue can avoid traffic accidents. Even if the time resolution is smaller than 1 s, drivers would still be likely involved in traffic accident. The solution is to make early judgment and decision making. This is another reason why we divided fatigue into mild fatigue and excessive fatigue. The distinction between mild fatigue and excessive fatigue is not only reasonable but also necessary. The trend of the experienced driver complexity curve is tantamount to the latter part [the bold part of the curve in Fig. 7(a)] of the inexperienced driver complexity curve. This result provides the basis for the distinction between the two kinds of drivers. It means that we can set different threshold values for the two kinds of drivers individually. For example, we can make the system give an alarm in a short period of time before experienced drivers are in excessive fatigue stage, but we may just need to make the system give an alarm when inexperienced drivers are in mild fatigue stage. Eye movements somehow reflect the state of the brain, and therefore, the results of EOG signal processing have a certain relationship with EEG. The two calculated fatigue times are exactly similar (see Fig. 4). Note that the fatigue state estimated by EMG is only mild fatigue within the range of the experimental time. The fatigue time obtained by EMG is the time when the subject gets lower tonus of the muscles. There is a significant increase in average EMG amplitude and a decrease in mean power frequency [6], [32]. That may cause the significant change in PP-ApEn [see Fig. 5(c)]. In this paper, the response and subjective assessment of the subject was used as a basis for verification. As shown in Tables I and II, the fatigue process reflected by the objective physiological signals is generally in accord with the fatigue process reflected by the response and subjective assessment. The estimated state of fatigue is the true fatigue experienced by the human subjects. As shown in Table I, the excessive fatigue time is 105.67 min, whereas the excessive fatigue time of the previous study where only the WE feature of the EEG signal is used is about 110.55 min [33]. According to Table II, the integrated approach of different entropies is more suitable for practical applications than just using the WE feature of the EEG signal. As shown in Table IV, the classification accuracy of ANN on the test data is about 96.5%–99.5%, whereas the accuracy of the previous study at which the power spectral density of DWT

176

IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 15, NO. 1, FEBRUARY 2014

of EEG was used is 95%–96% [31]. According to these results, the level of driver fatigue can be estimated better by using the method based on EEG, EOG, and EMG signals. Note that EEG signals have the key role in the driver fatigue estimation. As explained in the introduction, all the physical and mental activities associated with driving are reflected in EEG signals. However, some extracted features of EEG are not clear because of the impact of complex movement and decision making in the process of driving. To achieve a highaccuracy fatigue level, we used EEG, EMG, and EOG signals simultaneously and selected the best entropy feature for each signal. WE has a better effect on calculating the EOG fatigue time and reflecting mood swings (see Fig. 4); PP-SampEn has a better effect on calculating the EEG fatigue time (see Fig. 6); PP-ApEn has a better effect on calculating the EMG fatigue time (see Fig. 5). The inputs of the neural network classifier are determined by these.

VII. C ONCLUSION In this paper, a new real-time method for automated detection and identification of driver fatigue based on various entropy and complexity measures has been proposed. After driving for about 10 min, it will be known whether the subjects have extensive experience in driving by calculating the complexity of EEG ApEn. The real-time calculation results based on the entropies with sliding window are fed to the developed multilayer neural network classifier, and then, the driver fatigue level can be determined in real time.

R EFERENCES [1] M. Chipman and Y. L. Jin, “Drowsy drivers: The effect of light and circadian rhythm on crash occurrence,” Safety Sci., vol. 47, no. 10, pp. 1364– 1370, Dec. 2009. [2] A. Williamson, D. A. Lombardi, S. Folkard, J. Stutts, T. K. Courtney, and J. L. Connor, “The link between fatigue and safety,” Accid. Anal. Prev., vol. 432, pp. 498–515, Mar. 2011. [3] S. K. Lal, A. Craig, P. Boord, L. Kirkup, and H. Nguyen, “Development of an algorithm for an EEG-based driver fatigue countermeasure,” J. Safety Res., vol. 34, no. 3, pp. 321–328, Aug. 2003. [4] S. Kar, M. Bhagat, and A. Routray, “EEG signal analysis for the assessment and quantification of driver’s fatigue,” Transp. Res. F, vol. 13, no. 5, pp. 297–306, Sep. 2010. [5] K. Hyoki, M. Shigeta, N. Tsuno, Y. Kawamuro, and T. Kinoshita, “Improving the saccade peak velocity measurement for detecting fatigue,” J. Neurosci. Methods, vol. 187, no. 2, pp. 199–206, Mar. 2010. [6] T. Öberg, “Muscle fatigue and calibration of EMG measurements,” J. Electromyogr. Kinesiol., vol. 5, no. 4, pp. 239–243, Dec. 1995. [7] M. B. Kurt, N. Sezgin, M. Akin, G. Kirbas, and M. Bayram, “The ANNbased computing of drowsy level,” Expert Syst. Appl., vol. 36, no. 2, pp. 2534–2542, Mar. 2009. [8] S. K. Lal and A. Craig, “A critical review of the psychophysiology of driver’s fatigue,” Biol. Physiol., vol. 55, no. 3, pp. 173–194, Feb. 2001. [9] M. Sabeti, S. Katebi, and R. Boostani, “Entropy and complexity measures for EEG signal classification of schizophrenic and control participants,” Artif. Intell. Med., vol. 47, no. 3, pp. 263–274, Nov. 2009. [10] S. Janjarasjitt, M. S. Scher, and K. A. Loparo, “Nonlinear dynamical analysis of the neonatal EEG time series: The relationship between sleep state and complexity,” Clin. Neurophysiol., vol. 119, no. 8, pp. 1812– 1823, Aug. 2008. [11] W. Li, Q. C. He, X. M. Fan, and Z. M. Fei, “Evaluation of driver fatigue on two channels of EEG data,” Neurosci. Lett., vol. 506, no. 2, pp. 235–239, Jan. 2012.

[12] M. Patel, S. K. L. Lal, D. Kavanagh, and P. Rossiter, “Applying neural network analysis on heart rate variability data to assess,” Expert Syst. Appl., vol. 38, no. 6, pp. 7235–7242, Jun. 2011. [13] T. Takahashi, R. Y. Cho, T. Mizuno, M. Kikuchi, T. Murata, K. Takahashi, and Y. Wada, “Antipsychotics reverse abnormal EEG complexity in drugnaive schizophrenia: A multiscale entropy analysis,” NeuroImage, vol. 51, no. 1, pp. 173–182, May 2010. [14] C. E. Shannon, “A mathematical theory of communication,” Bell Syst. Tech. J., vol. 27, pp. 379–423, Jul./Oct. 1948. [15] S. M. Pincus and A. L. Goldberger, “Physiological time-series analysis: What does regularity quantify?” Am. J. Physiol., vol. 266, pt. 2, no. 4, pp. 1643–1656, Apr. 1994. [16] J. S. Richman and J. R. Moorman, “Physiological time-series analysis using approximate entropy and sample entropy,” Am. J. Physiol. Heart Circ. Physiol, vol. 278, no. 6, pp. 2039–2049, Jun. 2000. [17] A. Lempel and J. Ziv, “On the complexity of finite sequence,” IEEE Trans. Inf. Theory, vol. 22, no. 1, pp. 75–81, Jan. 1976. [18] Z. Z. Luo and M. Cao, “EEG feature analysis of motor imagery based on Lempel–Ziv complexity at multi-scale,” Chin. J. Sensors Actuators, vol. 24, pp. 1033–1037, Jul. 2011. [19] A. Yildiz, M. Akin, M. Poyraz, and G. Kirbas, “Application of adaptive neuro-fuzzy inference system for vigilance level estimation by using wavelet-entropy feature extraction,” Expert Syst. Appl., vol. 36, no. 4, pp. 7390–7399, May 2009. [20] S. M. Pincus, “Approximate entropy as a measure of system complexity,” in Proc. Natl. Acad. Sci., Mar. 1991, vol. 88, no. 6, pp. 2297–2301. [21] C. Zhang, H. Wang, and R. Fu, “Driver fatigue analysis based on EEG alpha rhythms,” in Proc. Int. Conf. Adv. Construction Machinery Veh. Eng., Changchun, China, 2013. [22] C. Zhang, H. Wang, H. Wang, and M. H. Wu, “EEG-based expert system using complexity measures and probability density function control in alpha sub-band,” Integr. Comput.-Aided Eng., to be published. [23] F. Q. Xue, “Study on evolutionary RBF neural network classifier,” Ph.D. dissertation, Dept. Info. Eng., Info. Eng. Univ., Zhengzhou, China, 2009. [24] M. Karpenko and N. Sepehri, “Neural network classifiers applied to condition monitoring of a pneumatic process valve actuator,” Eng. Appl. Artif. Intell., vol. 15, no. 3/4, pp. 273–283, Jun.–Aug. 2002. [25] A. Subasia and E. Ercelebi, “Classification of EEG signals using neural network and logistic regression,” Comput. Methods Programs in Biomed., vol. 78, no. 2, pp. 87–99, May 2005. [26] S. Dreiseitl and L. Ohno-Machado, “Logistic regression and artificial neural network classification models: A methodology review,” J. Biomed. Inform, vol. 35, no. 5/6, pp. 352–359, Oct. 2002. [27] I. A. Basheer and M. Hajmeer, “Artificial neural networks: Fundamentals, computing, design, and application,” J. Microbiol. Methods, vol. 43, no. 1, pp. 3–31, Dec. 2000. [28] B. B. Chaudhuri and U. Bhattacharya, “Efficient training and improved performance of multilayer perceptron in pattern classification,” Neurocomputing, vol. 34, no. 1–4, pp. 11–27, Sep. 2000. [29] L. Fausett, Fundamentals of Neural Networks Architectures, Algorithms, and Applications. Englewood Cliffs, NJ, USA: Prentice-Hall, 1994. [30] A. Bradley, “The use of the area under the ROC curve in the evaluation of machine learning algorithms,” Pattern Recog., vol. 30, no. 7, pp. 1145– 1159, Jul. 1997. [31] M. K. Kiymik, M. Akin, and A. Subasi, “Automatic recognition of alertness level by using wavelet transform and artificial neural network,” J. Neurosci. Methods, vol. 139, no. 2, pp. 231–240, Oct. 2004. [32] T. Öberg, “A validation of techniques using surface EMG signals from dynamic contractions to quantify muscle fatigue during repetitive tasks,” J. Electromyogr. Kinesiol., vol. 7, no. 2, pp. 131–139, Jun. 1997. [33] C. Zhang and H. Wang, “Real-time EEG signal analysis for driver fatigue detection based on wavelet entropy,” in Proc. Int. Conf. Med. Phys. Biomed. Eng., Qingdao, China, 2012.

Chi Zhang received the B.S. and M.S. degrees from Northeastern University Shenyang, China, in 2010 and 2012, respectively, where he is currently working toward the Ph.D. degree. His current research interests include biomedical signal processing, brain–computer interface, and artificial intelligence.

ZHANG et al.: AUTOMATED DETECTION OF DRIVER FATIGUE BASED ON ENTROPY AND COMPLEXITY MEASURES

Hong Wang received the B.S. and M.S. degrees from Northeastern University, Shenyang, China, in 1982 and 1986, respectively, and the Ph.D. degree from Otto von Guericke University of Magdeburg and Leibniz Institute for Neurobiology, Magdeburg, Germany, in 1998. She is currently a Professor with Northeastern University. She has authored or coauthored more than 120 papers in journals or conferences. Her research interests include brain–machine interface and electroencephalogram-based brain cognitive science.

177

Rongrong Fu received the Bachelor’s and Master’s degrees in measuring and control technologies and instruments from Liaoning University, Shenyang, China, in 2008 and 2011, respectively. She is currently working toward the Ph.D. degree in mechatronic engineering at the School of Mechanical Engineering and Automation, Northeastern University, Shenyang, China. Her current research interests include electroencephalogram signal processing, driver fatigue, and brain–computer interface.