An Image-based Mouth Switch for People with Severe Disabilities

66 Recent Patents on Computer Science 2012, 5, 66-71 An Image-based Mouth Switch for People with Severe Disabilities Mu-Chun Su1*, Chin-Yen Yeh1, Yi...
Author: Byron Manning
1 downloads 3 Views 445KB Size
66

Recent Patents on Computer Science 2012, 5, 66-71

An Image-based Mouth Switch for People with Severe Disabilities Mu-Chun Su1*, Chin-Yen Yeh1, Yi-Zeng Hsieh1, Shih-Chieh Lin1 and Pa-Chun Wang2 1

Department of Computer Science & Information Engineering, National Central University, Taiwan, R.O.C.; 2Cathay General Hospital, Taiwan, R.O.C Received: September 24, 2011; Accepted: December 22, 2011; Revised: December 26, 2011

Abstract: This study reports the design of a low-cost image-based mouth switch for the people with severe disabilities. Via this mouth switch and a communication aid, people with severe disabilities can manipulate computers and communicate with other people. Two experiments were conducted to test the performance of the mouth switch. Experimental results showed that this switch could help people manipulate computers simply by opening or closing the mouth. In addition, this article also presented some relevant patents about the human-computer interfaces for disabled people.

Keywords: Assistive technology, human computer interfaces, mouth switch, people with disabilities. 1. INTRODUCTION In the recent decades, various new technology products are rapidly emerging. Without any doubt, computers are one of the most influential technology products. With the growing use of computers, the quality and the lifestyle of our lives and even the whole society are dramatically changing. Unfortunately, conventional computer interfaces (e.g., mouse and keyboard) are designed for people who can dexterously manipulate computers with fingers. Some disabled people who cannot manually manipulate conventional computer interfaces; therefore, cannot enjoy the benefits provided by computers as able-bodied people on equal term. Apparently, how to design more user-friendly interfaces for disabled people to replace traditional computer input devices becomes a challenging and demanding task. The sip and puff switch is a commercially available and popular choice for many disabled people who have limited or no motor capability to operate switch-based devices (e.g., computer); however, it may be not the best choice for some types of disabled people (e.g., serious amyotrophic lateral sclerosis (ALS) patients). In addition, the sip and puff switch may suffer from the sanitation problem for some disabled people. Recently, advances in hardware and software have led to assistive technology systems of every variety which allow people with disabilities to use their limited voluntary motions to access computers, communicate with family and friends, and control home appliances, etc. While some of these systems were based on the head movements (e.g., sensor-based systems [1-4], vision-based systems [5, 6]), some systems were based on the body movements [7, 8]. In these years, more and more eye-based systems were developed [917]. Some of these systems were based on eye movements [9-14], and some of them were based on eye blinks [15-17]. Several different approaches have been proposed to develop *Address correspondence to this author at the Department of Computer Science & Information Engineering, National Central University, Taiwan, R.O.C.; Tel: +886-3-4227151; Ext: 35201; Fax: +886-3-4222681; E-mail: [email protected]

1874-4796/12 $100.00+.00

tongue-based systems for disabled people [18-20]. With the advance of digital signal processing technology, many braincomputer interfaces (BCIs) have also been proposed to improve the life quality of the disabled people [21-22]. Recently, several BCI products (e.g., Intendix [23], MCTOS brain switch [24]) have been commercially available for users. Each approach has its own advantages, disadvantages and limitations. For example, while in some systems people have to wear accessories on body (e.g., sensors, electrodes, etc), some systems are non-invasive because they are visionbased. Some head-movement-based systems are not applicable to serious ALS patients because they cannot voluntarily move their heads. To a certain extent, economic factor may hinder users from adopting an effective assistive system. In addition, the sanitation problem may hinder people from adopting some assistive systems which require users to hold a straw or adhere to a sensor into mouths. In one of our previous works [25], the performance of the eye tracking system had been evaluated by several ALS patients in a Hospital. Some ALS patients reported that they usually felt tired after they operated the interfaces for a period of time. This motivated us to propose a new kind of non-invasive interface. The goal of this paper is to present a low-cost image-based switch called a “mouth switch”. With the proposed switch incorporated with the communication aid, ALS people are able to use their barely limited voluntary motions such as mouth for communications, manipulating computers, and controlling home appliances. The remaining of this paper is organized as follows. In Section 2, the review of recent patents is given. In Section 3, the proposed switch will be described. The communication aid will be briefly introduced in Section 4. Section 5 presents the experimental results. Finally, Section 6 concludes the paper. 2. REVIEW OF RECENT PATTERNS A mouth-operated computer input device was proposed in US 7768499 [26]. It includes a mouthpiece to be posi© 2012 Bentham Science Publishers

An Image-based Mouth Switch for People

tioned in a user's mouth, and an optical device carried by the mouthpiece for emitting light from a surface area. Then the optical device controls movement of a cursor on a computer display based upon movement of the user's tongue across the surface area. Head pose assessment methods and systems were proposed in US 7844086 [27]. They can effectively assess a user's face and head pose such that a computer or like device can track the user's attention towards a display device(s) so that the region of the display or graphical user interface, the user is turned towards, can be automatically executed. A tongue-operated input device for control of electronic systems was proposed in US 7995031 [28]. In this patent, a method and apparatus for tongue-operated control of an electronic system were presented. A mouth switch which can change signals through the simple operation of holding the levers with a mouth for a multifunction operation microscope was proposed in the patent US 7375880 [29]. A mouth-operated computer input device and associated methods were proposed in the patent US 7768499 [30]. They include a mouthpiece to be positioned in a user's mouth and an optical device for emitting light towards the user's tongue. They control movements of a cursor on a computer display based upon movement of the user's tongue across the surface area.

Recent Patents on Computer Science, 2012, Vol. 5, No. 1

67

learnability, operability, personal acceptability, physical comfort, physical security, portability, securability, and supplier repairability. Two more recent articles attributed decisions to use or not to use an assistive device to some influential factors such as functional needs, price, comfort with use, etc [37, 38]. In addition, Yousefi et al. proposed a valuable reference model about how to evaluate a newly developed assistive device [39]. The aforementioned discussions provide not only selection criteria but also guides for future improvements or designs. 3. THE IMAGE-BASED MOUTH SWITCH Usually, a seriously disabled person may either lie in a bed or sit on a wheelchair so that the orientation and location of the camera have to be appropriately adjusted to be able to capture his or her mouth image. Therefore, our system adopts a low-cost R-1 battle snake-one web cam which is placed in front of a computer monitor as shown in Fig. (1). The camera supplies 15 color images of size 640480 per second. To achieve real-time performance, the mouth opening detection algorithm processes only 320240 pixels in gray level at an average 30 frames per second.

The US patent 20100013758 [31] proposed a ‘human interface device (HID)”. A user may wear the radio frequency human interface device on a body portion and move the body portion over any surface to provide inputs to a computer. A mouth activated input device for an electronically responsive device was proposed in US patent 20040164881 [32]. It includes an elongated tubular body having an exterior surface, a first end and a second end. Either a bite switch, a sip and puff switch or both are incorporated into the body. An apparatus for interfacing with a control object apparatus consisted of an image input unit, a gesture recognition unit, a control unit, and a gesture information display unit that was proposed in US patent 7844921 [33].

Fig. (1). The vision-based mouth switch which is implemented by a low-cost web camera.

The flowchart is shown in Fig. (2). The main steps of the mouth opening detection algorithm are summarized as follows:

In the US patent 7921364 [34], a computer user interface with sound was proposed to move a user interface pointer based at least partially upon the spectrum of frequencies and energies. A “wearable electromyography-based controller” which consisted of a plurality of Electromyography (EMG) sensors was proposed in US patent 20090326406 [35]. It provides a wired or wireless human-computer interface for interacting with computing systems via electrical signals generated by specific movement of the user's muscles. The most important criteria for evaluating an assistive device is whether it meets the needs of the disabled people. Batavia et al. have already identified 17 factors that consumers consider in determining whether a device meets their needs [36]. These 17 factors are affordability, compatibility, consumer repairability, dependability, durability, ease of assembly, ease of maintenance, effectiveness, flexibility,

Fig. (2). The flowchart of the mouth opening detection algorithm.

68 Recent Patents on Computer Science, 2012, Vol. 5, No. 1

Step 1. Initial Mouth Region Location The user is required to execute an initialization procedure to generate his or her mouth template before the use of the system. The system asks the user to open and then close his or her mouth for one time in order to acquire the opening and closure images, respectively. The frame differencing technique is employed to detect the initial mouth region. It is implemented by the computation of the difference between the opening and closure images on the basis of pixel by pixel. Step 2. Template Generating After the initial mouth region has been located, a bounding box will be automatically moved to encircle the initial mouth region to generate the template of the mouth. Then the width and the length of the bounding box, W and L, and the size of the box, S, are stored and used as the features of the “template” of the mouth to determine the possible opening of the mouth region in the next image frame.

Su et al.

access computers, communicate with other people, control home appliances, etc. The proposed communication aid is a modified version of the aid introduced in one of our previous works [16]. The aid which dichotomizes daily living necessities into 7 groups was proposed for people with severe disabilities (e.g., Voiced Messages, Typing, Home Appliance Control, Help, A/V Entertainments, Web Surfing, Messages) as shown in Fig. (4). In addition to the seven selections, another two selections, Suspend and Exit, are two available options. Most of the selections’ functionalities are selfevident. We did not optimize the aid because we tried to make the communication aid as simple as possible so that the user can intuitively use the aid without a tedious pre-training procedure.

Step 3. Mouth Opening Detection The frame differencing technique is employed to detect the moving object region during two consecutive frames. The detected moving pixels are then processed by the erosion and dilation morphological operators in order to remove some isolated pixels and some small sized region. The 8connectivity labeling algorithm is used to locate the remaining connected regions. If a region with the width/length ratio and the size similar as the template of the mouth, then a possible mouth opening is claimed to be detected. From our many simulations, we found that a candidate region with the width/length ratio (L / W ± 1) and the size (1 ± 0.1)S could be claimed to be a mouth region. A true mouth opening is detected if the mouth region is continually detected by several consecutive frames (i.e., 3 frames in our prototype system). Figure 3 shows an example of detection of the mouth from an image.

Fig. (3). The image-based mouth switch. (a) The closed mouth. (b) The opened mouth. (c) The detected mouth region.

Step 4. Issuing Clicking Signals If a complete mouth opening and closure is detected within a scanning time interval (e.g., 2 seconds in our prototype system) then a mouse clicking signal is issued to the computer to execute some kinds of functionalities. Otherwise, a new frame is processed to detect whether a mouth opening is detected. 4. THE COMMUNICATION AID The proposed mouth switch incorporated with a communication aid provides disabled people with the ability to

Fig. (4). The first layer of the communication aid in the English version.

The proposed switch can be incorporated with the communication aid to allow people with severe disability to access computers and communicate with others. The communication aid sequentially scans through these nine selections on the row by row basis. The user opens and then closes his/her mouth when his/her desired row is highlighted in red color. Then the aid scans through each selection in that row and waits for the switch signal issued by the user. If there is no mouse clicking signal issued in two complete scans at the present layer then the aid will automatically jump back to the upper layer and start to scan the selections at the upper layer. An example is shown in Fig. (5). Once the user subsequently selects the four selections, “Body”, “Head”, “Ear”, and “Ache”, the aid will automatically output the voice signal “My ear aches”. Basically, the contents of the “Voiced Messages” selection are edited and organized according to the suggestions of “Taiwan Motor Neuron Disease Association”. This kind of communication provides the ALS patient with the opportunity of expressing his or her desires, feelings, uncomfortableness in the body, etc. 5. EXPERIMENTAL RESULTS 5.1. Mouth Detection Experiment The experiment was conducted to test whether the mouth opening detection algorithm can successfully detect mouth openings under different conditions. Six subjects were asked

An Image-based Mouth Switch for People

Recent Patents on Computer Science, 2012, Vol. 5, No. 1

69

Fig. (5). An example of its subsequent selections of the “Voice Messages” selection to express “My ears ache”.

to open his/her mouth and each one was recorded in two different lighting conditions. We collected a data set consisting of 120 image sequences taken under varying lighting conditions. The sequences were manually examined offline to determine when a mouth opening occured. The total number of mouth opening in these images sequences was 120. The experimental result shows that the correct detection rate could reach 96.7% success rate. Furthermore, experiments were conducted to test whether head movements would influence the correct recognition rates of the algorithm. Each subject was asked to move his or her head horizontally or vertically while opening his or her mouth simultaneously. There were 48 image sequences in total. The error rate incurred by moving head horizontally was 0%; however, the error rate incurred by moving head vertically was 8.3%. It indicated that horizontal head movements would not degrade the performance of the mouth detection algorithm.

each subject repeated opening and closure of his or her mouth for three times before he or she started to type the word. Some subjects repeated the typing task for three times, some subjects completed the typing task for only one time. Therefore, there were in total 33 completed records. The program organizes the alphabets into 6 rows (shown in Fig. (6)). In this on-screen scanning keyboard, it took 2 seconds for a scan. The word “welcome” requires a total of 43 scans, therefore, it took at least 86 seconds to complete typing the word without any detection error. On average each subject took 110.52 seconds to complete the typing task. It indicated that some mouth openings were misdetected so the program took time to jump back to previous layers. Among 33 completed records, there were 7 best records which showed 86 seconds (i.e., no error) and the worst record showed 338 seconds. The subject with the 338 seconds record took 238 seconds on average in his three trials. As for the most dexterous subject, he took 94 seconds on average from the three trials to finish the typing task.

5.2. Text Typing Experiment In this experiment, 15 subjects were asked to use the mouth switch to type the word “welcome” via the on-screen scanning keyboard provided by the communication aid. Before the typing task, each subject was simply informed to use his or her mouth as a switch for running the on-screen scanning keyboard. Without any tedious pre-training procedure,

6. CURRENT & FUTURE DEVELOPMENTS In this paper, a low-cost image-based mouth switch is presented. The proposed mouth switch provides disabled people with a simple and intuitive approach to access the computer by simply opening/closing the mouth. The

70 Recent Patents on Computer Science, 2012, Vol. 5, No. 1

Su et al. [6]

[7]

[8]

[9] [10] [11]

Fig. (6). The scanning keyboard.

proposed mouth switch is an appealing interface because it is a low-cost and non-invasive interface. In addition, compared to the commercially available sip and puff switch, our image-based mouth switch does not encounter the sanitary problem. The performance of the mouth switch may degrade due to several factors such as head movements and a sudden and dramatic change of lighting conditions. To improve the robustness of the mouth switch, the template used in the mouth detection procedure may need to be updated after a certain time interval. In addition, we will consider providing the word prediction functionality, increasing scanning speed, and re-configuring an efficient and effective scanning layout to enhance the typing rate in the future. Finally, more field tests and formal evaluations will be conducted in hospitals in the near future. 7. ACKNOWLEDGEMENTS

[12] [13]

[14]

[15] [16]

[17]

[18] [19]

[20]

This paper was partly supported by the CGH-NCU Joint Research Foundation Project number: 99 CGH-NCU A4 and the National Science Council, Taiwan, R.O.C, under NSC992911-I-008-100, 98-2221-E-008-094-MY3, and the NSC100-2631 -S-008-001.

[21]

8. CONFLICT OF INTEREST

[22]

The authors claimed no conflict of interest. REFERENCES [1]

[2] [3]

[4]

[5]

Y.L. Chen, F.T. Tang, W.H. Chang, M.K. Wong, Y.Y. Shih, and T.S. Kuo, “The new design of an infrared-controlled humancomputer interface for the disabled,” IEEE Trans. Rehab Eng., vol. 7, pp. 474-81, 1999. D.G. Evans, R. Drew, and P. Blenkhorn, “Controlling mouse pointer position using an infrared head-operated joystick,” IEEE Trans. Rehabil Eng., vol. 8, pp. 107-17, 2000. M.C. Su, Y.H. Lee, C.H. Wu, S.Y. Su, and Y.X. Zhao, “Two lowcost human computer interfaces for people with severe disabilities,” Biomed Engin – Appl, Basis & Commun., vol. 16, pp. 344-9, 2004. S. Kim, M. Park, S. Anumas, and J. Yoo, “Head mouse system based on gyro- and opto-sensors,” In International Conference on Biomed. Eng. Informatics., 2010, pp. 1503-6. M.C. Su, S.Y. Su, and G.-D. Chen, “A low cost vision-based human-computer interface for people with severe disabilities,” Biomed Engin-Appl, Basis, & Commun, vol. 17, pp. 284-92, 2005.

[23] [24]

[25] [26] [27] [28] [29] [30]

M. Tanaka, M. Hara, Y. Morito, T. Sakamoto, K. Miyamori, S. Moromugi, and T. Ishimatsu “Vision based pointing device with slight body movement,” in Int. Conf. Control, Automat Syst, 2008, pp. 1990-1993. R.B. Reilly and M.J. O’Malley, “Adaptive noncontact gesturebased system for augmentative communication,” IEEE Trans. Rehabil Eng, vol. 7, no. 2, pp. 174-182, 1999. M. Betke, J. Gips, and P. Fleming, “The camera mouse: Visual tracking of body feature to provide computer access for people with severe disabilities,” IEEE Trans. Neural. Syst. Rehabil Eng., vol. 10, no. 1, pp. 1-10, 2002. G. Norris and E. Wilson, “The eye mouse: an eye communication device”, in IEEE 23rd Northeast Bioengin. Conf., 1997, pp. 66-67. C. H. Morimoto, D. Koons, A. Amit, M. Flickner, and S. Zhai, “Keeping an eye for HCI,” in Proc. XII Brazilian Symp. Computer Graphics and Image Processing, 1999, pp. 171-176. P. DiMattia, F.X. Curran, and J. Gips, An eye control teaching device for students without language expressive capacity: Eagleeyes, Edwin Mellen Pr, Lampeter, U.K., 2001. D. Kumar and E. Poole, “Classification of EOG for human computer interface,” in the Second Joint EMBS/BMES Conference, 2002, pp. 23-26. M.C. Su, K.C. Wang, and G.-D. Chen, “An eye tracking system and its application in aids for people with severe disabilities,” Biomed Eng-Appl, Basis, Commun, vol. 18, pp. 319-27, 2006. L.J. G. Vazquez, M.A. Minor, and A.J.H. Sossa, “Low cost human computer interface voluntary eye movement as communication system for disabled people with limited movements,” Pan American Health Care Exchanges (PAHCE), pp. 165-70, 2011. K. Grauman, M. Betke, J. Gips, and G.R. Bradski, “Communication via eye blinks- detection and duration analysis in real time,” in Proceedings CVPR 2001, pp. I-1010-1017. M.C. Su, C.Y. Yeh, S.C. Lin, P.C. Wang, and S.M. Hou, “An Implementation of an Eye-Blink-based Communication Aid for People with Severe Disabilities,” in Proceedings International Conference Audio, Langage, Image Process, 2008, pp. 351-356. A. Krolak and P. Strumillo, “Vision-based eye blink monitoring system for human-computer interfacing,” in Conference on Human System Interactions, 2008, pp. 994-998. W. Nutt, C. Arlanch, S. Nigg, and G. Staufert, “Tongue-mouse for quadriplegics,” J. Micromech. Microeng., vol. 8, pp. 155-7, 1998. N.S. Lotte and A. Struijk, “A tongue based control for disabled people,” in 10th International Conference of Computers Helping People with Special Needs, 2006, pp. 913-918. G. Krishnamurthy and M. Ghovanloo, “Tongue drive: A tongue operated magnetic sensor based wireless assistive technology for people with severe disabilities,” in IEEE International Symposium on Circuits and Systems, 2006, pp. 5551-5554. F. Nijboer, E.W. Sellers, J. Mellinger, M.A. Jordan, T. Matuz, A. Furdea, S. Halder , U. Mochty , D. J. Krusienski , T. M. Vaughan , J. R. Wolpaw , N. Birbaumer, and A. Kübler, “A P300-based brain–computer interface for people with amyotrophic lateral sclerosis,” Clin. Neurophysiol., pp. 1909-1916, 2008. F. Cincotti, D. Mattia, F. Aloise, S. Bufalari, G. Schalk, and G. Oriolo, “Non-invasive brain–computer interface system: towards its application as assistive technology,” Brain Res. Bull., vol. 75, pp. 796-803, 2008. Intendix. Available at: http://singularityhub.com/2010/03/07/ intendix-the-brain-computer-interface-goes-commercial-video/. (Accessed on: Nov. 6, 2011) MCTOS brain switch. Available at: http://www.synapseadaptive.com/switches/brain_%20switch.htm. (Accessed on: Nov. 6, 2011) K.C. Chiu, “The development of an eye-tracking system and its applications,” M. S. thesis (in Chinese), National Central University, Taiwan, 2006. C.R. Sturtz, “Mouth-operated computer input device and associated methods,” U. S. Patent 7768499, 2010. Y. Hu, L. Zhang, M. Li, and H. J.Zhang, “Head pose assessment methods and systems,” U. S. Patent 7844086, 2008. K. Manal, “Tongue-operated input device for control of electronic systems,” U. S. Patent 7995031, 2005. K. Nakamura, “Mouth switch mechanism for operation microscope,” U. S. Patent 7375880, 2008. C.R. Sturtz, “A mouth-operated computer input device and associated methods,” U. S. Patent 7768499, 2010.

An Image-based Mouth Switch for People [31] [32]

[33] [34]

[35]

A. Biswas and B. Keshavachar, “Human interface device (HID),” U. S. Patent 20100013758, 2010. M. Loun, K. Kelvin, P. F. Mok, A. Ruefli, M. V. Fedorak, L. Ming, H. Edmond, and C. Kirchen “A mouth activated input device for an electronically responsive device,” U. S. Patent 20040164881, 2004. T. Ike, Y. Taniguchi, R. Okada, N. Kishikawa, K. Yokoi, M. Yuasa, and B. Stenger, “Interface apparatus and interface method,” U. S. Patent 7844921, 2011. N. Ramirez, “Controlling a computer user interface with sound including defining ranges of audio frequencies; associating each range with a user interface command; receiving as input from a user a sound,” U. S. Patent 7921364, 2011. D. Tan, S. Saponas, D. Morris, and J.Turner, “A wearable electromyography-based controller,” U. S. Patent 20090326406, 2009.

Recent Patents on Computer Science, 2012, Vol. 5, No. 1 [36]

[37] [38] [39]

71

A. I. Batavia and G. S. Hammer, “Toward the development of consumer-based criteria for the evaluation of assistive devices,” J. Rehabil. Res. Develop., vol. 27, pp. 425-436, 1990. A.M. Cook and S.M. Hussey, Assistive technologies: principal and Practice. 3rd ed. New York: Mosby, 2007. M.J. Scherer, Living n the state of stuck: how assistive technology impacts the lives of people with disabilities, 4th e. Boston, MA: Brookline Book, 2005. B. Yousefi, X. Huo, E. Veledar, and M. Ghovanloo, “Quantitative and comparative assessment of learning in a tongue-operated computer input device,” IEEE Trans. Inf. Technol. Biomed., vol. 15, pp. 747-757, 2011.

Suggest Documents