An Implementation of an Eye-blink-based Communication Aid for People with Severe Disabilities

An Implementation of an Eye-blink-based Communication Aid for People with Severe Disabilities Muchun Su1,2, Chinyen Yeh1, Shihchieh Lin1, Pachun Wang3...
4 downloads 2 Views 310KB Size
An Implementation of an Eye-blink-based Communication Aid for People with Severe Disabilities Muchun Su1,2, Chinyen Yeh1, Shihchieh Lin1, Pachun Wang3, Shawmin Hou3 1 Department of Computer Science & Information Engineering, National Central University, Taiwan, R.O.C. 2 Graduate Institute of Biomedical Engineering, National Central University, Taiwan, R.O.C. 3 Cathay General Hospital, Taiwan, R.O.C. E-mail: [email protected] motion-based systems may provide an alternative option for people with severe disabilities who only retain the ability to move their eyes. There are several different ways to track the eye movements, such as refection of light [9]-[13] and electrooculographic potential (EOG) [10], [14]-[19], etc. Among so many useful assistive technology systems, the “camera mouse” system [20] and the “Blink Link” [21] deserve to be particularly mentioned. The camera mouse system tracks some small section of a user’s facial features (e.g., nose, lip, and the whole eye, etc) with a video camera and translates them into the movements of the mouse pointer on the screen. By dwelling in the desired screen area for a certain amount of time, the user may make a selection or issue a mouse click. The experiences with the camera mouse system were very encouraging. They showed that the system could successfully provide computer access for people with severe disabilities. However, the eye feature has not been used effectively with the camera mouse system in their reported work at that time. Grauman et al. proposed the Blink Link system which enables communication using eye blink patterns to provide an alternate input modality to allow people with severe disabilities to access a computer [21]. A very high success rate in almost real-time was reported; however, the system was imposed by some restrictions. Once the open eye template becomes out of date for some reasons, the system may give faulty outputs. In [22], Bhaskar et al. even pointed out that the Blink Link system suffers from several disadvantages. For example, it requires offline training for different depths from the camera for the computation of the distance. Furthermore, changing camera positions requires the whole system to be retrained. In our previous work, a vision-based “Head Mouse” system [23] and an “eye mouse” [24] have been

Abstract This paper presents an implementation of a lowcost vision-based computer interface which allows people with severe disabilities to use eye blinks to access computers and communicate with other persons. Our communication aid requires only one low-cost web camera and a personal computer. Several experiments were conducted to test the performance of the proposed eye-blink-based communication aid.

1. Introduction Computers have been dramatically changing our lifestyles, livelihoods, and even the whole society. These kinds of changes benefit some groups in our society but unavoidably create new barriers to a disadvantage minority such as people with physical disabilities who cannot manually access computers with dexterity as able-bodies people do. Therefore, in recent years, there has been an effort to design alternative interfaces for people with disabilities to replace traditional computer input devices such as keyboard and mouse. Assistive technology systems of every variety have been proposed and even commercialized to allow people with disabilities to use their limited voluntary motions to communicate with family and friends, access computers, and control TVs and air conditioners, etc [1]-[8]. For some people with severe disabilities, an extreme disability such as amyotrophic lateral sclerosis (ALS) or severe cerebral palsy deprives them of the use of their limbs and even facial muscles. Owing to this kind of extreme disability, many available popular assistive technology systems are not helpful to them. Under this circumstance, eye-

978-1-4244-1724-7/08/$25.00 ©2008 IEEE

351

ICALIP2008

proposed to allow people with disabilities to use their head movements or eye movements to manipulate computers. The goal of this paper is to present an implementation of a non-instructive eye-blink-based communication aid for the severe disabilities such as ALS. With the communication aid, ALS people are able to use their limited voluntary motions such as eye blinks for communications, manipulating computers, and controlling home appliances (e.g., TV and air conditioner, etc). In this communication aid, the pattern matching technique and optical flow are integrated to detect eye blinks. The remaining of this paper is organized as follows. In Section 2, the system design will be described. Section 3 introduces the experimental results. Finally, Section 4 concludes the paper.

Fig. 1 The first layer of the communication aid in the English version

2. The proposed communication aid Our proposed communication aid for people with severe disabilities dichotomizes daily living necessities into 7 groups (e.g., Voiced Messages, Typing, Home Appliance Control, Help, A/V Entertainments, Web Surfing, Messages) as shown in Fig. 1. In addition to the seven selections, another two selections, Suspend and Exit, are another two available options. Most of the selections’ functionalities are self-evident. For example, the A/V entertainments selection allows the user to choose either listening music or watching movies. By adding more subsequent selections at the deeper layers as shown in Fig. 2, the user can easily adjust the volume or switch to another song or movie. As for the “Voiced Messages” selection, some voiced messages in common use can be issued by the use via subsequently blinking the wanted selections. An example is shown in Fig. 3. Once the user subsequently select the four selections, Body, Head, Scalp, and Itch, the aid will automatically output the voice signal “My scalp itches”. The contents of the “Voiced Messages” selection are edited and organized according to the suggestions of “Taiwan Motor Neuron Disease Association”. This kind of communications provides the ALS patient with the opportunity of expressing movements that he or she wants to do, feelings, uncomfortableness in body, etc. The communication aid sequentially scans through these nine selections on the row by row basis. The user blinks when his or her desired row is highlighted in red color. Then the aid scans through each selection in that row and waits for the blink signal issued by the user. If there is no detected blink in two complete scans at the present layer then the aid will automatically jump back to the upper layer and start to scan the selections at the upper layer.

Fig. 2 The “A/V Entertainments” selection and its subsequent selections

2.1 Hardware The system consists of a 3.2GHz Pentium 4 PC with the Windows XP operating system and a low-cost R-1 battle snake-one web cam. The Web camera is placed in front of the computer monitor. The camera supplies 15 color images of size 640 × 480 per second. To achieve real-time performance, the eye blink detection algorithm processes only 320×240 pixels in gray level at an average 30 frames per second.

2.2 The eye blink detection algorithm The applications of eye blink detection and analysis are widely varied from communication aids for the disabled, driver drowsiness detection, cognitive engagement, operator attentiveness monitoring, etc [25]-[33]. Among so many different approaches, the frame differencing technique is the most popular method used for eye blink detection [21]. Some

352

present image frame and used as the first four “templates” (as shown in Fig. 4) to determine the possible position of the eye region in the next image frame. Since the illumination condition and the distance between the camera and the user may vary from time to time during the operation procedure, the templates should be updated in accordance with the environmental changes. Otherwise, the templates may become out of date and wrong eye regions may be located. In fact, an intuitive and simple approach for generating the template for the eye region is to use a box around the center of the working eye region. However, from many experimental results, we found that the performance of the simple template was not as high as we expected. One possible reason is that the updated template may gradually lose the representative of the pupil and then detect wrong regions. That is why we adopted the four templates since good results could be expected.

Fig. 3 The “voice messages” selection and its subsequent selections

Step 3. Eye tracking These four templates generated in the previous step are used for eye tracking. Since we hope the response time can be minimized we use the pattern matching technique to locate the eye regions in the subsequent frames. We use the templates to search for the eye region in the subsequent frame. For each template, the template is matched with the test sub-images. Let t (m, n) and s (m, n) represent the template and the test sub-image in the current frame, respectively. Template matching is performed using the following correlation coefficient ∑ ∑ (t (m, n) − t m )( s(m, n) − s m )

Fig. 4 The four small templates around the pupil approaches adopt optical flow [32]-[33]. Each approach has its own considerations, limitations, and advantages. Some approaches could achieve a very high successful rate such as in [21] but some approaches could only reported 65% success rate such as in [32]. Of courses, the price paid for achieving a high success rate may be restrictions in operational environments. For example, the Blink Link system could detect eye blinks with a success rate of 96.5% in almost real-time; however, it suffers from several restrictions as mentioned in the previous section. In this paper, we integrate the pattern matching technique and optical flow to detect eye blinks. It involves in four steps.

γ =

m

(

∑∑ m

n

where t m and

n

t (m, n) − t m )(

∑ ∑ s ( m, n ) − s m ) m

(1)

n

s m are the averages of the t (m, n) and

s (m, n) . A higher correlation coefficient γ indicates the two images have similar brightness patterns. The template with the highest correlation coefficient among four templates is chosen to be the candidate of the eye region. If the highest correlation coefficient is larger than a pre-specified threshold, θ cc , then an eye region is claimed to be detected and four templates around the pupil are generated for the next frame. Otherwise, go to next step and use optical flow to detect whether the user blinks.

Step 1. Initial eye region location The system asks the use to blink his or her eye when the user tries to start to use the communication aid so that the eye region could be effectively located by the simple frame differencing method. Step 2. Templates generating After the initial eye region has been located, a circle with an appropriate size will be automatically moved to encircle the pupil. Then four small rectangle-shaped templates around the pupil are cropped out of the

Step 4. Blink detection Optical flow computation is based on two assumptions: (1) the brightness of any feature point is constant over time and (2) nearby points in the image

353

move in a similar manner. A pyramidal implementation of a hierarchical optical flow method [34]-[35] is used to automatically track the 25 anchor points uniformly distributed in each rectangle-shaped template. If the average moving length of the 25 anchor points is larger than a threshold, θ of , then a

3.2 Typing test The subjects were asked to use a scanning spelling program to type “ci lab”. The program organizes the alphabets into 3 groups. Each group contains 6 rows. In this spelling program, it takes 2 seconds for a scan. Six strokes require 18 selections. Without any error in detecting blinks, it totally requires 98 seconds to complete the typing task. The average typing time across the subjects was 114.5 seconds. It indicates that some blinks were miss-detected so the program took time to jump back to previous layers. The experimental result shows that the performance rate can reach 94.75% success rate.

blink motion is claimed to be detected. One complete eye blink involves two motions: open-closed followed closed-open. An example of the average motion length across time is depicted in Fig. 5 where two consecutive peaks represent an eye blink. Moreover, the number of frames lying between the two peaks may serve as an indication of a voluntary blink or not. A prolonged blink with more than three frames between the two consecutive peaks indicates a voluntary blink. If the average motion length is less than the threshold, θ of ,

4. Conclusions

then a blink is not detected. Therefore, we need to go back to the previous step to search the eye region in a larger region. If the frequency of miss-detected of the eye region is higher than a threshold, θ md , then the

In this paper, an implementation of a low-cost eyeblink-based communication aid for ALS patients is presented. Experimental results show that it can be used to manipulate the computer for people via blinking eyes.

system will automatically go to the first step to ask the user to voluntarily blink to generate another four new templates. voluntary blink

voluntary blink

involuntary blink

5. Acknowledgements This paper was partly supported by the 96CGHNCU-A3, the National Science Council, Taiwan, R.O.C, under the NSC-96-2221-E-008-017, the NSC96-2752-E-008-002-PAE, the NSC-96-2524-S-008002, and the NSC-96-2422-H-008-001.

involuntary blink

6. References Fig. 5 An example of the average motion length across time and two consecutive peaks indicate a complete eye blink

3. Experimental results

[1]

W. J. Perkins and B. F. Stenning, “Control units for operation of computers by severely physically handicapped persons,” J. Med Eng. Technol., vol. 10, no. 1, 1986, pp. 21-23.

[2]

O. Takami, N. Irie, C. Kang, T. Ishimatsu, and T. Ochiai, “Computer interface to use head movement for handicapped people,” in Proc. IEEE TENCON’96, Digital Signal Processing Applications, vol. 1, 1996, pp. 468-472.

[3]

D. G. Evans, R. Drew, and P. Blenkhorn, ”Controlling mouse pointer position using an infrared head-operated joystick,” IEEE Trans. on Rehabilitation Engineering, vol. 8, no. 1, 2000, pp. 107-117.

[4]

Y. L. Chen, F. T. Tang, W. H. Chang, M.K. Wong, Y. Y. Shih, and T. S. Kuo, “The new design of an infrared-controlled human-computer interface for the disabled,” IEEE Trans. on Rehabilitation Engineering, vol. 7, Dec. 1999, pp. 474-481.

3.1 Eye blink detection test The experiment was conducted to test whether the eye blink detection algorithm can successfully detect eye blinks under different conditions. Four subjects were asked to blink and each one was recorded in two different lighting conditions. We collected a data set consisting of 8 image sequences taken under varying lighting conditions. The sequences were manually examined offline to determine when and how many eye blinks happened. The total numbers of frames and eye blinks in these testing sequences were 560 and 40, respectively. The experimental result shows that the performance rate could reach 97.5% success rate.

354

[5]

[6]

[7]

[8]

[9]

R. B. Reilly and M. J. O’Malley, “Adaptive noncontact gesture-based system for augmentative communication,” IEEE Trans. on Rehabilitation Engineering, vol. 7, no. 2, 1999, pp. 174-182.

[18] Y. Tomita, Y. Igarashi, S. Honda, N. Matsuo, “ Electro-Oculography Mouse for Amyotrophic Lateral Sclerosis Patients ”, in the 18th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 1996, pp. 1780 -1781.

M. C. Su, W. C. Cheng, P. Z. Chang, L. Z. Chang, Y. W. Huang, and C. Y. Tew, “A simple and inexpensive telephone dialing aid for the disabled,” in IEE Computing & Control Engineering Journal, vol. 11, no. 2, April 2000, pp. 73-78.

[19] K. S. Park and K. T. Lee, ”Eye-controlled human/computer interface using the line-of-sight and the intentional blink”, Computer Engineering, Vol. 30, No.3, pp. 463-473, 1996.

M. C. Su, C. Y. Chen, S. Y. Su, C. H. Chou, H. F. Hsiu, and Y. C. Wang, “A Portable Communication Aid for Deaf-Blind People,” in IEE Computing & Control Engineering Journal, vol. 12, no. 1, February 2001, pp. 37-43.

[20] M. Betke, J. Gips, and p. Fleming, “The camera mouse: visual tracking of body feature to provide computer access for people with severe disabilities,” IEEE Trans. on Neural Systems and Rehabilitation Engineering, vol. 10, no. 1, 2002, pp. 1-10.

M. C. Su, Y. H. Lee, C. H. Wu, S. Y. Su, and Y. X. Zhao, “Two Low-Cost Human Computer Interfaces for People with Severe Disabilities,” Biomedical Engineering – Applications, Basis & Communications, Vol. 16, No. 6, Dec. 25, 2004, pp. 344-349.

[21] K. Grauman, M. Betke, J. Gips, and G. R. Bradski, “Communication via eye blinks- detection and duration analysis in real time,” Proc. CVPR 2001, 2001, pp. I1010-1017.

Eye-Trace System, Permobil Meditech AB, Timra, Sweden, http://www.algonet.se/~eyetrace.

[22] T. N. Bhaskar, F. T. Keat, S. Ranganath, and Y. V. Venkatesh, “Blink detection and eye tracking for eye location,” TENCON 2003, pp. 821-824.

[10] L. Young and D. Sheena, “Survey of eye movement recording methods,” Behav. Res. Meth. Instrum., vol. 7, no. 5, 1975, pp.397-429.

[23] M. C. Su, S. –Y. Su, and G. –D. Chen, “A low cost vision-based human-computer interface for people with severe disabilities,” in Biomedical EngineeringApplications, Basis, & Communications, Vol. 17, No. 6, 2005, pp. 284-292.

[11] T. Hutchinson, K. P. White Jr., W. N. Martin, K. C. Reichert, and L. A. Frey, “Human-computer interaction using eye-gaze input,” IEEE Trans. Systems, Man, Cybernetics, vol. 19, no. 6, 1989, pp. 1527-1533.

[24] M. C. Su, S. –Y. Su, and G. –D. Chen, “A low cost vision-based human-computer interface for people with sevee disabilities,” in Biomedical EngineeringApplications, Basis, & Communications, Vol. 17, No. 6, 2005, pp. 284-292.

[12] G. A. Rinard, R. W. Mateson, R. W. Quine, and R. S. Tegtmeyer, “An infrared system for determining ocular position,” ISA Trans., vol. 19, no. 4, 1980, pp. 3-6. [13] C. H. Morimoto, D. Koons, A. Amit, M. Flickner, and S. Zhai, “Keeping an eye for HCI,” in Proc. XII Brazilian Symp. Computer Graphics and Image Processing, 1999, pp. 171-176.

[25] T. Brandt, R. Stemmer, and A. Rakotonirainy, “Affordable visual driver monitoring system for fatigue and monotony,” in Proc. IEEE lntemational Conference on Systems, Man and Cybemetics, vol. 7, 2004, pp.6451-6456.

[14] D. Kumar and E. Poole, “Classification of EOG for human computer interface,” in the Second Joint EMBS/BMES Conference, vol. 1, Oct. 2002, pp. 23-26.

[26] K. Fukuda, J. A. Stern, T. B. Brown, and M. B. Russo, “Cognition, blinks, eye-movements, and papillary movements during performance of a running memory task,” Aviation, Space, and Environmental Medicine, vol. 76, July 2005, pp. C75–C85.

[15] J. R. LaCourse and F. C. Hludik Jr., “An eye movement communication-control system for the disabled,” IEEE Trans. on Biomedical Engineering, vol. 37, no. 12, 1990, pp.1215-1220.

[27] M. F. Funada, S. P. Ninomija, S. Suzuki, K. Idogawa, Y. Yam, and H. Ide, “On an image processing of eye blinking to monitor awakening levels of human beings,” in Proc. 18th Annual International Conference of the Engineering in Medicine and Biology Society, vol. 3, 1996, pp. 966–967.

[16] P. DiMattia, F. X. Curran, and J. Gips, An Eye Control Teaching Device for Students Without Language Expressive Capacity: EagleEes, Lampeter, U.K.: Edwin Mellen, 2001.

[28] Q. Ji, Z. Zhu, and P. Lan, “Real-time nonintrusive monitoring and prediction of driver fatigue,” IEEE

[17] G. Norris, E. Wilson, “The eye mouse: an eye communication device”, IEEE 23rd Northeast Bioengineering Conference, May 1997, pp. 66-67.

355

Transactions on Vehicular Technology, vol. 53, no. 4, July 2004, pp. 1052–1068.

[32] M. J. Black, D. J. Fleet, Y. Yacoob, “A framework for modeling appearance change in image sequences,” International Conference on Computer Vision, 1998.

[29] H. Lim and V. K. Singh, “Design of healthcare system for disable person using eye blinking,” in Proc. Fourth Annual ACIS International Conference on Computer and Information Science, 2005, pp. 551–555.

[33] T. N. Bhaskar, F. T. Keat, S. Ranganath, and Y. V. Venkatesh, “Blink detection and eye tracking for eye location,” Conference on Convergent Technologies for Asia-Pacific Region, vol. 2, 2003, pp. 821-824.

[30] P. Smith, M. Shah, and N. da Vitoria Lobo, “Determining driver visual attention with one camera,” IEEE Transactions on Intelligent Transportation Systems, vol. 4, no. 4, December 2003, pp. 205–218.

[34] B. D. Lucas and T. Kanade. “An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 8, 1986, pp. 565-593.

[31] R. Heishman and Z. Duric, “Using image flow to detect eye blinks in color videos,” IEEE Workshop on Applications of Computer Vision, 2007.

[35] J. Y.Bouguet, “Pyramidal Implementation of the Lucas Kanade Feature Tracker Description of the algorithm,” Intel Corporation Microprocessor Research Labs.

356

Suggest Documents