Characteristics of Brain Wave Changes by Affective Pictures Ruoyu Du1 and Hyo Jong Lee1, 2 Division of Computer Science and Engineering, Chonbuk National University, Jeonju, Korea 2 Center for Advanced Image & Information Technology, Chonbuk National University, Jeonju, Korea 1

Abstract - Emotion status influences important areas in our daily lives. Many researchers reported successful emotion classifications. The aim of this study is to find out the neurophysiological characteristics of the brain waves while affective pictures elucidate emotion. Four healthy college students volunteered the stimulus experiment with the standard IAPS affective pictures. All brain waves showed active pattern over the frontal and parietal lobes. The significances of emotion change were found frontal lobe and central gyri area for Alpha band. Beta band also showed significance for emotion change around parietal lobe. This study revealed only Alpha and Beta waves changed significantly at limited location due to changed emotional status. Keywords: Electroencephalogram (EEG); emotion; valencearousal model; IAPS; ICA;

1

Introduction

Every human being expresses emotion daily lives. Emotions are an especially interesting phenomenon as a result of the huge impact they have on humans on a daily basis. Emotions play an important role in communication and affect our behaviors. Because of the importance of emotions, many researchers have investigated various psychological and physiological phenomena. An electroencephalogram (EEG) has been recognized as a useful tool because of its high time resolution and the direct information from a brain invasively. It also detects the abnormal brain waves or electrical activity of the brain. Many scientists reported various methods to retrieve emotion-related information from EEG signals with the advanced biomedical signal processing technology. This has resulted in the ability to detect a variety of psychological and physiological states. EEG based emotion research is a challenging field within the area of biomedical signal processing. Recently several researches were performed to understand human emotion. Nie[1] et al. investigated the relationship between EEG signals and human emotions. They used EEG signals to classify either positive or negative emotions. A support vector machine (SVM) was applied to the extracted features from original EEG data. They reported 87% of accuracy between

positive and negative emotion. Lin [2] et al. investigated EEG-based emotion elicited by auditory stimulus. Using the music-induced emotional responses, a comparative study was conducted to find out if hierarchical binary classifiers work better than nonhierarchical method. Kwon and Lee [3] used EEG signals which are stimulated while watching movie clips. Changes in alpha and gamma power have been interpreted to indicate differential valence pattern related to the frontal lobes. Liu [4] et al. proposed real-time algorithm of quantification of basic emotions using Arousal-Valence emotion model. An EEG-based web-enable music player was implemented, which can display the music according to the user’s current emotion states. Since an emotion elicitation procedure occurs in our mind by watching images or listening music, it is safe to assume the EEG signals reflect emotion-related brain activity. The signals provided by this procedure can then be processed to train and test the system of EEG based emotion recognition. However, these researches have not investigated neurophysiological phenomena arising inside a brain. To neuroscientists their feature data simply reflect unrecognizable numbers. A fuzzy approached training method or a SVM were trained with known emotion states and predict a subject’s emotion condition based on trained numbers. The high classification rate means that randomly selected feature data were trained well. However, this does not mean that neurophysiological process, which occurred inside a brain, has been understood clearly. The commonly practiced method for EEG of emotion research consists of three steps. First, EEG data are recorded and collected under predefined stimulus by using the biomedical related machine. Secord, preprocessing techniques are applied to signals to remove artifact noises, such as EOG, ECG and muscle movement signals. Third, each brain wave is extracted and analyzed in terms of power or patterns. The purpose of this paper is to find out neurophysiological characteristics of brain activity under emotional affection. For example, feature points can be extracted from alpha wave, beta wave or gamma wave in frontal lobes because all types of waves are activated under emotional condition. Through this study, the neurophysiological information will be able to distinguish which brain wave is prevalent and which wave will show significance during emotion changes.

2 2.1

Related Theory Emotions

Two main approaches are commonly used to recognize emotions: the taxonomy approach and the dimensional approach. The taxonomy approach, in some cases is also known as the evolutionary approach [5]. The taxonomy approach derives from the non-cognitive theories, as well as from the somatic approaches. This approach considers emotions to be discrete and, therefore, characterization of each emotion is independent with regards of the others. Moreover, these discrete emotions would be useful responses to specific environmental situations, as a result of the evolution. Thus, this is also called the evolutionary approach. The dimensional approach, derived from the cognitive theories, characterizes the state of mind of a person in terms of dimensions. Common dimensions are the valence of emotion (positive or negative) and the arousal level of emotion (active or passive). This 2D valence-arousal (V-A) emotion model is more expressible and general than the discrete emotion approaches. In this paper, the V-A emotion model is used to classify emotion criterion. And the subjective assessment of emotion is represented from 0 to 9 as the level of evaluation, which is shown in Figure 1.

potentials are known as brain waves, and the entire record is called an EEG [6]. Intensity of EEG recording range from 0 to 200 microvolt on the surface of the scalp, and their frequency ranges from once every few seconds to 50 or more per second. The characteristics of the waves are dependent on the degree of activity in respective parts of the cerebral cortex. The waves change markedly between the states of emotions. Much of the time, the brain waves are irregular, and no specific pattern can be discerned in the EEG. There are mainly five types of Brain waves: Delta waves (0.5-4 Hz) which are considered to be related to the deep sleep [7] in the adults or premature babies. It is usually found in the frontal region of brain in adults and posterior region in children. A common Theta wave (4-8 Hz) which occurs in children and adults when they are in emotional stress or they have deep midline disorders. It is found in parietal and occipital region. Another type of theta waves is named frontal midline theta. The theta waves exist during the various tasks which need the correlation of the increased mental effort and sustained concentration [8]. Alpha wave (8-13 Hz), which occurs in quiet resting state but not sleep, is found in the occipital region. Alpha waves can reflect the relaxation level a person is having. They are also believed to be responsible for the movement related brain activity. Another role of Alpha rhythms is to handle a perceptual processing, memory tasks, and emotions [7]. Beta wave (13-30 Hz) occurs in active and busy concentration or anxious thinking state. It is found in the frontal and parietal region and is related to the concentration level of people [5]. An increase in a beta power may reflect the increase of the arousal level of an emotional state [8]. Gamma wave (30-100 Hz) which occurs in certain cognitive or motor functions. It is often used for diagnosis of the certain brain illness [7].

2.3

Fig. 1. Valence-Arousal Model

2.2

EEG and Brain waves

Electrical recordings from the surface of the brain, or even from the outer surface of the head, demonstrate that there are continuous electrical activities in the brain. Both the intensity and the patterns of this electrical activity depend on the level of excitation of different parts of the brain resulting from sleep, wakefulness, or brain diseases such as epilepsy or even psychoses. The undulations in the recorded electrical

Emotion in the Brain

Stimuli are transmitted into the brain at the brain stem. The limbic system around the brain stem is responsible for initial emotional interpretation of these signals from the autonomic nervous system. This part of the brain has also been found important for motivation and memory functions. Although motivation and memory also have their influence on the reaction to emotional stimuli, the rest of the text will focus on the limbic structures. They are also responsible for emotional reactions. The hypothalamus is responsible for processing the incoming signals and triggering the corresponding visceral physiological effects, like a raised heart rate or galvanic skin response [9]. From the hypothalamus the stimuli information is passed on to the amygdala, which is important for learning to stimulate emotional response (reward/fear) and evaluate the new stimulus by comparing their past experiences. The amygdala is thought to be important for processing emotion. However, since it is an underlying structure like the rest of the limbic system, it cannot be detected directly in recordings from the scalp. The amygdala is connected to the temporal and

prefrontal cortices, which is considered to be the way visceral sensations are interpreted cognitively, resulting in a consciously experienced feeling of an emotion [9]. The temporal lobe is essential for hearing, language and emotion, and also plays an important role in memory. The prefrontal lobe (directly behind the forehead) is involved in the highest level of functioning. It is responsible for cognitive, emotional and motivational processes. The prefrontal lobe is part of the frontal cortex, which is allegedly known as the emotional control center and to even determine personality. It is involved in, among others, judgment and social behavior. These functions are very much effective based on the experience of emotions [10].

Figure 2, FMS Falk Minow Services, Herrsching-Breitbrunn, Germany). In this research, eighteen electrodes (Fp1, Fp2, F3, F4, Fz, F7, F8, C3, C4, Cz, T7, T8, P3, P4, P7, P8, O1, and O2) were inserted to record EEG signals using the Easy Cap which refer to Figure 2. The electrooculogram (EOG) was recorded from one additional electrodes placed below the outer canthi of left eye. Impedances were kept below 5 kΩ. Cardiac activity was recorded with an electrode from the left outer of neck, ECG drawn in Figure 2. An average reference was used. Brain Vision Recorder was used to record the data (0.5-70 Hz, 500 samples per second). Subjects were instructed to remain still and to blink or move their eyes and body as little as possible during the recording periods.

3

3.3

3.1

Materials and Methods Subjects

In this experiment, four healthy males in the age group of 23-25 years old were recruited as subjects. They are all right-handed and have correct visions. All of the subjects were undergraduate students of the same institution and were informed about the purpose of this research. Once the consent forms were filled-up, the subjects were given a simple introduction about the research work and stages of experiment.

3.2

EEG Signal Acquisition

EEG signals were collected using the 10/20 internationally recognized placement system shown in Figure 2. This system is based on the relationship of various position of electrode placed on scalp and the underlying area of cerebral cortex [11].

Experimental Construction

Procedure

and

Stimuli

The whole experiment was designed to induct emotion within the valence (positive / approach versus negative / withdrawal) and arousal (calm versus excited) space. These two dimensions are a subset of the three-dimensional representation [12] [13] for collecting affective ratings for the IAPS. Figure 3 shows the relationships between arousal and valence of IAPS images. To make clear distinction among emotions, five affective states were selected: low arousal-low valence (LA_LV), low arousal-high valence (LA_HV), high arousal-high valence (HA_HV), high arousal-low valence (HA_LV) and middle arousal-middle valence (MA_MV). On the basis of these ratings, 35 pictures (7 pictures x 5 states) were selected from uniformly distributed clusters along the valence and arousal axes.

Fig. 2. The montage used in this research based on 10-20 system of electrode placement.

Fig. 3. Scatterplot of valence and arousal ratings (1–9 scales) for all available International Affective Picture System images. The circles denote the images selected in this study.

The EEG signals were recorded using a Brain Vision amplifier system (BrainProducts, Germany). Silver-silverchloride-electrodes (Ag/AgCl) were used in association with the “Easy Cap System” (International 10-20 system shown in

Figure 3 also shows the location of the pictures finally selected at the arousal-valence domain with the circles. During the experiment, the selected pictures were projected randomly for 4s following another 4s for resetting emotion

with a blurred image. Due to its unknown emotional status before the projection of the first picture and after the projection of the last picture, a fixation mark (cross) was projected for eight seconds in the middle of the screen to attract the sight of the subject. Figure 4 shows the timing diagram of this experiment. The total time of collecting EEG recording in this experiment was 296 seconds. After all of the pictures were projected, a Self-Assessment Mannequin (SAM) [14] procedure took place. The EEG signals from each subject were recorded during the whole projection phase. It is important to distinguish between emotional stimulus and experienced emotion. IAPS stimuli are associated with “standard ratings”, but the same picture may not induct the same level of arousal and valence. For this reason, the subjects were asked to rate their own emotional experience while being presented each stimulus. Each subject was told about the importance of these ratings, with particular emphasis on the importance to rate how the subject actually felt while viewing each picture.

Fig. 4. Timing diagram for five different emotions. Each category has seven pictures totalling 35 pictures.

3.4

Preprocessing

An open source tool box named EEGLAB provided by SCCN lab [15], running under the cross platform of MATLAB environment (Mathworks, Inc.) is used for both preprocessing and analysis of the EEG data. It includes data collection functions, channel and event information management and data visualization tools, such as scrolling, scalp map and dipole model plotting and multi-trial ERPimage plots. Main preprocessing features are artifact rejection, filtering, epoch selection and averaging signals. It also provides high level analysis functions, such as PCA and ICA. After the acquisition phase, the EEG data was at first referenced to Cz electrode while importing the EEG data files to the EEGLAB. Then the channel locations were imported for getting information about the recording electrodes which is necessary for plotting EEG scalp maps or to estimate source locations for data components. The data was then high-pass filtered with lower cut-off frequency of 4Hz and low-pass

filtered with cutoff frequency of 50Hz. This band-pass filtering of continuous EEG data using linear FIR filter eliminated the power line noise, EMG and EOG artifacts.

3.5

Artifact Detection and Removal via ICA

ICA (Independent Component Analysis) algorithms have proven capable of separating artifacts and neural signals generated from EEG whose EEG contributions, across the training data, are maximally independent of one another. ICA is widely used in the EEG research community to detect and remove eye, muscle, and line noise artifacts and also to separate biologically plausible brain sources whose activity patterns are distinctly linked to behavioral phenomena. EEGLAB contains an automated version of the Infomax ICA algorithm with several enhancements. ICA finds a coordinate frame in which the data projections have minimal temporal overlap. The core mathematical concept of ICA is to minimize the mutual information among the data projections or maximize their joint entropy [15]. In this paper, ICA was applied to reduce the total number of features from 2000 (500Hz sampling rate, for 4 seconds), to first 20 components. For each of the number of independent components within this research, the classification procedure was applied, as the optimal number was unknown at this point. ICA applied to a matrix of EEG scalp data finds an unmixing matrix of weights W which linearly decomposes the multichannel data into a sum of maximally temporally independent and spatially fixed components u  Wx . The rows of the output matrix u are courses of activation of the ICA components. These components account for artifacts, stimulus and response locked events and spontaneous EEG activity. The columns of the inverse matrix W 1 give the relative projection strengths of the respective components at each of the scalp sensors. This is the process of ICA decomposition of the data into maximally temporally independent processes, each with its distinct time series and scalp map. These scalp maps of projection strengths provide evidence for the components’ physiological origin (e.g. ocular activity projects mainly to frontal sites). Selected components can be projected back onto the scalp using the relation x0  W 1u0 , where u0 is the matrix u with irrelevant components set to zero. Thereby brain signals accounted for by the selected components can be obtained in true polarity and amplitudes [16]. Eye movement artifacts result from the contamination of the EEG by the electrooculogram (EOG), a potential produced by movement of the eye or eyelid. Several methods have been proposed for removing ocular artifacts from the EEG, most of which make use of a separate EOG record. There are two main types of eye movement artifacts, those due to blinks and those due to saccadic movements [17-19]. Blink artifacts are due to contact of the eyelid with the cornea which alters ocular conductance. The influence of blink artifacts on recording electrodes decreases rapidly with

distance from the eyes. Saccade artifacts arise from changes in orientation of the retina-corneal dipole. The cornea of the eye is positively charged relative to the retina. Rotation of the retina-corneal axis results in changes in electrical potential. The saccadic influence decreases much slower and shows a typical pattern of polarity difference between contra-lateral sites. ICA has already been used successfully for blind source separation of EEG data. Application of ICA to ERPs include artifact detection and removal [18] [20] [21] as well as analysis of event-related response averages [16] [22]. Application of ICA to single-trial ERPs is more recent [20] [23] [24]. In single-trial EEG analysis, the rows of the input matrix x are EEG and EOG signals recorded at different electrodes and the columns are measurements at different time points. There also have the other two types of artifact discussed here is due to cardiac and muscle activity. Compared to normal EEG activity, muscle artifact is characterized by high frequencies (over 15Hz) and often by high amplitude. Because of the cardiac signal have a regular wave. ICA can separate those artifacts easily and remove the interference [25]. Either conscious or unconscious muscle activity produces an electric potential called electromyogram (EMG). Muscle artifacts can be classified according to their spread in space (broad or localized) and time (transient or permanent) [26] which also can be removed by using ICA.

3.6

Fig. 5. Mean PSD values of Alpha wave in the frontal region with standard deviation based on the different emotion states.

Data Analyses and Statistics

The power spectral density estimates were logtransformed (using the base 10 logarithm) in order to normalize their distribution. Spectral estimates were averaged within alpha (8-13 Hz), beta (13-30 Hz) and gamma (30-50 Hz) bands. For each epoch of each subject the spectral power data present the magnitude of signals at measurement points with colors. EEGLAB also shows the power spectrum of the brain model at the chosen frequency. Therefore, it is easy to know the activated parts on the brain during each event. In this paper, the power spectrum at the alpha frequency band, the beta frequency band and the gamma frequency band were plotted respectively to study the scalp distribution of power spectral density during stimuli. A probability of p