PORTABLE HAPTIC FEEDBACK FOR TRAINING AND REHABILITATION
A DISSERTATION SUBMITTED TO THE DEPARTMENT OF MECHANICAL ENGINEERING AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
Li Jiang March 2009
c Copyright by Li Jiang 2009
All Rights Reserved
ii
I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy.
(Mark R. Cutkosky) Principal Adviser
I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy.
(Larry J. Leifer)
I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy.
(H.F. Machiel Van der Loos)
Approved for the University Committee on Graduate Studies.
iii
Abstract The work presented in this dissertation focuses on the development and evaluation of portable haptic devices in procedure training in a virtual reality environment and neurological rehabilitation. Due to the cost, size and complexity of the grounded haptic devices, their use is still limited to research labs and is not popularized. Compared with grounded devices, portable haptic devices don’t have those limitations. However, portable devices cannot provide haptic feedback as realistic as grounded devices do. In my research, I specifically focuses on determining the effectiveness of providing portable haptic feedback to virtual reality procedure training and neurological rehabilitation where normal haptic feedback channels are distorted or absent. In the experiment for virtual reality procedure training, subjects participated in simulated exercises for clearing a damaged building, implemented using a modified commercial video game engine and USB-compatible force and vibration feedback devices. With the addition of portable haptic feedback, subjects made fewer procedural errors and completed some tasks more rapidly. Initial experiments on neurological rehabilitation were conducted with stroke patients. A novel hand-opening device was invented to help stroke patients in opening their impaired hands. Portable vibrotactile haptic feedback was provided in an experiment of controlling their grasp force in two approaches: amplitude based feedback(ABF) and event-cue based feedback(ECF). Results show that portable haptic feedback can improve subjects performance with either of the two approaches. Further extensive studies on portable haptic feedback for neurological rehabilitation were conduced with multiple sclerosis patients. First, a simple and low-cost
iv
system was invented. This system can help multiple sclerosis patients with asymmetric impairment to exert better grasp force control in manipulation tasks. The approach consists of measuring force vectors at the fingertips of the impaired hand, computing the force imbalance among the fingers and providing corresponding haptic signals to the fingers of the opposite hand. Tests conducted on 24 multiple sclerosis patients indicated that for those with mild impairment, slightly better results were obtained with an “event cue” feedback that alerted them when the grasp forces were staying outside of a desirable range. For patients with more severe impairment, better results were obtained by providing a proportional signal, in which the frequency and duty cycle of vibration pulses were correlated directly with the magnitudes of the fingertip forces. Post-test surveys of the patients indicated also that mildly impaired subjects preferred event-cue feedback and more severely impaired subjects preferred the proportional feedback. Experiment results from both virtual reality procedure training and neurological rehabilitation have shown that artificial haptic cues provided by portable haptic devices can improve people’s performance. Furthermore, it was observed that there is a correlation between the level of impairment and the type of feedback that is most effective. Severely impaired patients did better with proportional feedback; mildly impaired patients did better with occasional event cues.
v
Acknowledgments First and the foremost, I sincerely appreciate the guidance, encouragement and generous support provided by my advisor Professor Mark Cutkosky throughout my graduate study at Stanford. With his guidance and encouragement, I was able to discover my interest and develop my research. With his generous support in many ways, I was able to go through the good times and the rough spots in my studies. I could never have accomplished this task without his help. I am grateful to Professor Larry Leifer, for his support throughout my time at Stanford. I would also like to thank Professor Machiel Van der Loos for his suggestions on writing of this manuscript and his help on developing human subject protocols. I also wish to thank Professor Roope Raisamo, Dr Juhani Ruutiainen for inviting me to conduct research experiments in Finland and Deborah Kenney for helping me conduct survey at post-stroke survivors (REACH) as well as getting me in touch with potential experiment candidates for my stroke studies. A special thanks is due my colleagues and fellow graduate students, Karlin Bark, William Provancher, Trey McClung, Weston Griffin, Jason Wheeler, Yong-Lae Park, Sangbae Kim, Sean Bailey, Alan Asbeck, Jonathan Karpick, Sanjay Dastoor and Salomon Trujillo for the thoughtful discussions, mutual teaching and moral support during this time. Finally, I am grateful for the assistance and enthusiast support of my wife, Lanlan Tang. Financial support of this work was provided by National Science Foundation under SGER 0554188, Finnish Funding Agency for Technology and Innovation(Tekes), decision 40219/06 and Navy STTR N0014-03-M-0264.
vi
Contents Abstract
iv
Acknowledgments
vi
1 Introduction
1
1.1
Context and Motivation . . . . . . . . . . . . . . . . . . . . . . . . .
1
1.2
Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5
1.3
Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7
2 Background and Relevant Work
8
2.1
Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
8
2.2
VR Systems for Mobile Persons . . . . . . . . . . . . . . . . . . . . .
9
2.3
Wearable Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18
2.4
Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
19
3 Haptic Feedback for Virtual Reality Training 3.1
22
Experiment One . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24
3.1.1
Experiment One Setup . . . . . . . . . . . . . . . . . . . . . .
24
3.1.2
Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
26
3.1.3
Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
28
3.1.4
Metrics
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
29
3.1.5
Data Analysis and Results . . . . . . . . . . . . . . . . . . . .
30
3.1.6
Learning Rates . . . . . . . . . . . . . . . . . . . . . . . . . .
32
3.1.7
Experiment One Conclusion . . . . . . . . . . . . . . . . . . .
32
vii
3.2
3.3
Experiment Two . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
3.2.1
Experiment Two Setup . . . . . . . . . . . . . . . . . . . . . .
35
3.2.2
Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
38
3.2.3
Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39
3.2.4
Metrics
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
40
3.2.5
Data Analysis and Results . . . . . . . . . . . . . . . . . . . .
40
3.2.6
Experiment Two Conclusions . . . . . . . . . . . . . . . . . .
42
Discussions and Conclusions . . . . . . . . . . . . . . . . . . . . . . .
44
4 Enhancing Stroke Rehabilitation
45
4.1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
45
4.2
A Survey of Stroke Patients . . . . . . . . . . . . . . . . . . . . . . .
47
4.2.1
Survey Questions and Results . . . . . . . . . . . . . . . . . .
47
4.2.2
Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
48
Design and Development of a Portable Hand-Opening Device . . . . .
49
4.3.1
Design Goals . . . . . . . . . . . . . . . . . . . . . . . . . . .
49
4.3.2
Design Description . . . . . . . . . . . . . . . . . . . . . . . .
50
4.3.3
Device Validation . . . . . . . . . . . . . . . . . . . . . . . . .
62
Adding Haptic Feedback to a Hand-Opening Device . . . . . . . . . .
62
4.4.1
Pilot Experiment Procedure . . . . . . . . . . . . . . . . . . .
62
4.4.2
External Haptic Feedback . . . . . . . . . . . . . . . . . . . .
65
4.4.3
Experimental Data Analysis . . . . . . . . . . . . . . . . . . .
66
Summary and Conclusions . . . . . . . . . . . . . . . . . . . . . . . .
67
4.3
4.4
4.5
5 Haptic Aid for Multiple Sclerosis
69
5.1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
69
5.2
Hardware Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
71
5.3
Experiment Procedure . . . . . . . . . . . . . . . . . . . . . . . . . .
74
5.4
Force to Vibration Mapping . . . . . . . . . . . . . . . . . . . . . . .
76
5.4.1
Event cue feedback . . . . . . . . . . . . . . . . . . . . . . . .
77
5.4.2
Amplitude based feedback . . . . . . . . . . . . . . . . . . . .
78
5.4.3
Perception Test of Amplitude Based Feedback(ABF) Mode . .
78
viii
5.5
5.6
Experiment results . . . . . . . . . . . . . . . . . . . . . . . . . . . .
83
5.5.1
Force Control Metric . . . . . . . . . . . . . . . . . . . . . . .
83
5.5.2
Comparison with unimpaired subjects in a force balance task .
88
5.5.3
Failure Rates for MS patients . . . . . . . . . . . . . . . . . .
91
5.5.4
Task Completion Time for MS patients . . . . . . . . . . . . .
92
5.5.5
Experiment survey for MS patients . . . . . . . . . . . . . . .
95
Discussion and Conclusion . . . . . . . . . . . . . . . . . . . . . . . .
97
6 Conclusions
100
6.1
Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . . . 102
6.2
Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Bibliography
104
ix
List of Tables 3.1
Pattern of testing for subject groups (6 experienced and 6 inexperienced) 28
3.2
Average numbers of errors (failures to check rooms properly before entering) for inexperienced video game players. Four trials per subject per condition: A = no haptic feedback, B = joystick vibration feedback, C = joystick force feedback. . . . . . . . . . . . . . . . . . . . . . . .
3.3
30
Average numbers of errors (failures to check rooms properly before entering) for experienced video game players. Four trials per subject per condition: A = no haptic feedback, B = joystick vibration feedback, C = joystick force feedback. . . . . . . . . . . . . . . . . . . . . . . .
30
3.4
Sequences of obstacles encountered in the test corridors for Experiment 2 39
3.5
Sequences of corridors and trials, with or without haptic feedback, for eight subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
39
4.1
Sequences of haptic feedback modes for the 6 subjects . . . . . . . . .
64
5.1
Sequences of haptic feedback modes for the first 6 subject . . . . . . .
76
5.2
Question One results: Votes indicate how many subjects believed they did best with the corresponding mode; Correct votes are those that match the actual best mode for the same subjects.
. . . . . . . . . .
95
5.3
Preferences for ABF versus ECF modes . . . . . . . . . . . . . . . . .
96
5.4
Average helpfulness ratings for ABF and ECF modes (-3 = substantial hindrance, 0 = neutral, +3 = substantial help). . . . . . . . . . . . .
5.5
96
Average perceived ease of learning for ABF and ECF modes on a scale from -3 (quite difficult) to +3 (quite easy). x
. . . . . . . . . . . . . .
97
List of Figures 2.1
Individual Combatant Simulators . . . . . . . . . . . . . . . . . . . .
10
2.2
Avatars as part of a virtual team . . . . . . . . . . . . . . . . . . . .
11
2.3
Soldiers on Uniport and TreadPort . . . . . . . . . . . . . . . . . . .
12
2.4
Soldier on the Omni-Directional Treadmill . . . . . . . . . . . . . . .
13
2.5
The Sarcos Biport and The GaitMaster . . . . . . . . . . . . . . . . .
14
2.6
The Haptic Walker . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
2.7
The HapticGEAR . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
2.8
An overview of the multi-projector display D-vision and a user wearing the SPIDAR-H. Each hand has 4 cables to realize force feedback. . .
16
The ARCMIME system and MIME system . . . . . . . . . . . . . . .
17
2.10 A Whole Body Display . . . . . . . . . . . . . . . . . . . . . . . . . .
18
2.11 Vibrotactile display on arm, waist, shoulder and fingernail . . . . . .
19
2.12 Panasonic Stroke Rehabilitation Suit for upper extremity . . . . . . .
20
2.13 The Hand Motion Assist Robot . . . . . . . . . . . . . . . . . . . . .
21
3.1
Screenshot one of half life environment as used in experiment one. . .
24
3.2
Screenshot two of half life environment as used in experiment one. . .
26
3.3
Diagram of clipping effect used to approximate the collision force ex-
2.9
perienced when a user contacts a wall. . . . . . . . . . . . . . . . . .
27
3.4
The user must stop against the wall prior to entering a new room. . .
28
3.5
Map of a typical building layout in experiment one . . . . . . . . . .
29
3.6
Box plots showing the error statistics for experienced and inexperienced subjects in experiment one. . . . . . . . . . . . . . . . . . . . . . . .
xi
31
3.7
Error statistics for subjects as a function of trial number. There is a slight reduction in the average number of errors between trials 1 and 4. 33
3.8
Experiment setup side view: two vibration tactors were attached to the head of the subject and two vibration tactors were attached to the legs of the subject . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.9
35
Experiment setup front view: two vibration tactors were attached to the head of the subject and two vibration tactors were attached to the legs of the subject. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
36
3.10 Top view of the vibration tactor . . . . . . . . . . . . . . . . . . . . .
37
3.11 Side view of the vibration tactor . . . . . . . . . . . . . . . . . . . . .
37
3.12 Layout of the corridor in experiment two. . . . . . . . . . . . . . . . .
38
3.13 Obstacles (total reported) and number of obstacles correctly recorded, with or without haptic feedback. . . . . . . . . . . . . . . . . . . . . .
40
3.14 Histogram of the relative time required to complete a trial with haptics versus without haptics for the two speed trials. . . . . . . . . . . . . .
41
3.15 Histogram of the relative time required to complete a trial with haptics versus without haptics for the two memory trials. Although speed was not a goal in these trials, all but two subjects completed the task faster when haptic feedback was present. . . . . . . . . . . . . . . . . . . . .
42
4.1
Number of hours subjects spent on rehabiliation exercises daily . . . .
48
4.2
Impaired (left) and unimpaired (right) hands wearing their respective subsystems of the hand-opening device, shown in the closed (flexed) configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3
51
Impaired (left) and unimpaired (right) hands with their respective subsystems of the hand-opening device, shown in the opened configuration. 52
4.4
Shape Deposition Manufacturing cycle involving material additional removal and part embedding
. . . . . . . . . . . . . . . . . . . . . .
54
4.5
Top view of the hand-opening device for the impaired hand . . . . . .
55
4.6
Top view of hand-opening device when the hand is open . . . . . . .
55
xii
4.7
Finger Holder parametric CAD model. The marked dimensions are in millimeters and are adjusted to accommodate different finger sizes.
4.8
56
The tail section of the finger holder (left) prevents buckling of the metacarpophalangeal joint (right).
4.9
.
Top view of finger holder
. . . . . . . . . . . . . . . . . . .
57
. . . . . . . . . . . . . . . . . . . . . . . .
58
4.10 The skin protector and wrist brace. Red line represents the embedded metal plate that supports the palm. A positive angle between the palm and forearm, as shown in the figure, can make it easier for the user to grasp objects.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
58
4.11 Back support and cable guides. . . . . . . . . . . . . . . . . . . . . .
59
4.12 CAD rendering of cylinder and piston guide assembly . . . . . . . . .
60
4.13 Device for the unimpaired side, showing master cylinder, cable guides, protective sleeve and finger holders. . . . . . . . . . . . . . . . . . . .
61
4.14 Two different sizes of finger holders for the unimpaired hand. . . . . .
61
4.15 Main design objectives and the design features of the portable handopening device that contribute to them. . . . . . . . . . . . . . . . . .
63
4.16 CAD model of instrumented object consisting of a box with a threesided cover. The force sensors measure the forces between the cover and the base of the box, from which the normal component of the grasp force can be computed.
. . . . . . . . . . . . . . . . . . . . . . . . .
65
4.17 Grasping force data with three different haptic feedback modes. To reduce subject to subject variation, data are normalized by the average force without feedback (NHF condition). . . . . . . . . . . . . . . . .
66
4.18 Failure rate with three different haptic feedback modes. . . . . . . . .
67
5.1
System Hardware Diagram
71
5.2
Sensor plate with one cent coin. Three sensors are held in the base
5.3
. . . . . . . . . . . . . . . . . . . . . . .
plate, sensors are not wired in this figure to show a clear layout. . . .
72
Fingers with sensors attached. . . . . . . . . . . . . . . . . . . . . . .
72
xiii
5.4
Experiment setup: force sensors are attached on fingertips of subject’s right hand. Vibrotactile tactors are attached on the fingernails of subject’s left hand. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.5
Corrrelation between the force differences measure and results of the clinical nine-hole pin test. . . . . . . . . . . . . . . . . . . . . . . . .
5.6
75
Period length and duty cycle versus force, in the amplitude based mapping method (ABF). . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.7
73
79
Average perceived magnitude, normalized for each subject, corresponding to the periods and duty cycles in equation 5.1 and 5.2 associated with different force levels. Error bars show standard error. . . . . . .
5.8
80
The average minimum detectable difference between vibration stimuli corresponding to different nominal force values between 0.75 N and 3.75 N. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.9
82
Typical force plots with ABF (amplitude based feedback) mode and with NHF (no haptic feedback) mode
. . . . . . . . . . . . . . . . .
84
5.10 Sum of force differences, fs , under NHF(no feedback), ABF(amplitude) and ECF(event cue) feedback modes. Both ABF and ECF result in significantly smaller variations in force compared to NHF; no significant difference is found between ABF and ECF. (Data for each subject are normalized by the subject’s average value under NHF.) . . . . . . . .
85
5.11 Difference in percent improvement for ABS vs ECF with respect to NHF mode. Each point corresponds to one subject. The x coordinate in log scale shows subjects’ impairment level, IL . A line shows the best log fit to the data. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
86
5.12 Force variation, fs , for different feedback modes, with subjects divided into groups based on the degree of impairment. In both groups, the ABD and ECF feedback modes show improvement over the no-feedback (NHF) case. In the less-impaired group, the ECF mode was significantly better than ABF (p < 1 · 10−5 ); in the more impaired group the reverse was true(p < 1 · 10−5 ). . . . . . . . . . . . . . . . . . . . . . .
xiv
87
5.13 Sum of force differences of unimpaired subjects for the same task as MS subjects under the different feedback modes (NHF: no haptic feedback, ABF: amplitude based feedback, ECF: event cue feedback). Using a Bonferroni corrected T test, no significance was found between NHF and ABF or between ABF and ECF. However significance was found between NHF and ECF. (Data for each subject are normalized by the subject’s average value under no haptic feedback.)
. . . . . . . . . .
89
5.14 Sum of force differences for unimpaired subjects balancing forces while moving an object over a 30 cm high obstacle. Significance was found between each pair of feedback modes. (NHF: no haptic feedback, ABF: amplitude based feedback, ECF: event cue feedback). (Data for each subject are normalized by the subject’s average value under no haptic feedback.) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
90
5.15 The numbers of failures per subject for 24 MS patients under the three different feedback modes are statistically different. ABF provides the lowest failure rate. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
91
5.16 Task completion time under three different modes. Time was normalized by each subject’s average completion time in NHF mode. . . . .
93
5.17 Task completion time in ABF mode and ECF mode versus impairment level. Each dot or circle represents one subject. Best fit lines are plotted to the data.
. . . . . . . . . . . . . . . . . . . . . . . . . . .
93
5.18 Difference in completion time for ABS vs ECF with respect to NHF mode. Each circle represents one subject. The X coordinate shows subjects’ impairment level, IL . . . . . . . . . . . . . . . . . . . . . . .
94
5.19 Difference in completion time for ABS vs ECF with respect to NHF mode. Each circle represents one subject. The X coordinate shows subjects’ impairment level, IL . . . . . . . . . . . . . . . . . . . . . . .
xv
94
Chapter 1 Introduction 1.1
Context and Motivation
Imagine that you have just opened the door of a hotel room. The lights are off and the room is dark. What is the first thing that you do? Most probably you use your hand to explore the wall just inside the door, searching for a light switch. Mechanoreceptors in your hand send signals along your nerves to tell you when you have found the switch. After locating it, you continue to use your tactile senses to determine the type of switch and how to turn it on, combining the tactile information with prior knowledge that you have stored about switches. Similar feedback mechanisms are involved in most daily activities, often accompanied by sight and sound. We take this ability to integrate tactile information for exploration and manipulation for granted and rarely stop to consider how important it is until it is absent, whether due to a temporary condition, as when our fingers become numb from cold, or a long-term condition arising from a pathology. However, for patients suffering from diseases such as stroke, multiple sclerosis and diabetes, the loss of sensation in the peripheral nerves is a severe problem. Not only is it difficult or impossible to manipulate objects securely but the loss of sensation can lead to dangerous situations and injury. For example, it is easy for multiple sclerosis patients to drop things since they do not have a good idea of how much force they are applying, and a stroke patient can have his impaired hand damaged by a wheelchair without noticing it. 1
CHAPTER 1. INTRODUCTION
2
Over the last two decades, there has been steady progress in the design and control of haptic feedback devices that can artificially recreate the forces and/or vibrations that our mechanoreceptors sense as we interact with objects and devices in the world. The applications of this technology include realistic videogames with force feedback, teleoperation (e.g., for medical robotics) and numerous human/machine interaction systems ranging from cell phones to automobiles, in which haptic feedback provides an extra communication channel. Haptic feedback has also been applied to help people learn new activities, and to re-learn how to perform everyday manipulation and locomotion tasks following a stroke or surgery. As the applications of haptic feedback have expanded, there has been an increasing interest in portable and even wearable haptic display technologies. The first haptic feedback devices were relatively large desktop machines and cost thousands of dollars each. These devices employed servo motors and required amplifiers and power supplies capable of supplying and controlling tens of watts of power, making it difficult to use them with batteries. Recently, haptic feedback has been introduced to the touch screens of cell phones (e.g. Samsung Anycall Haptic) and other small, mass-produced consumer devices for which there is premium on minimizing cost, weight and power consumption. The most common, and simplest, application of haptic feedback is the ubiquitous use of pager motors in cell phones, pagers and videogame controllers. However, other modalities that may be suitable for low-cost, portable displays have also been demonstrated, at least in a research setting. Examples include skin stretch [37, 3], contact temperature [106], contact location [77] and friction [78] displays. The range and fidelity of haptic sensations that pager motors, touch-sensitive screens and similar devices can produce is quite limited in comparison to the effects that the larger desktop based devices can create. Nonetheless, they can provide useful cues about events such as an incoming call or that a user’s finger has passed over a virtual button on a display. One of the most significant distinctions between the portable displays and the desktop devices has to do with whether the effects are grounded with respect to a stationary reference frame. Desktop devices can produce grounded forces: when a user touches a stationary object in a virtual world, the force applied to the user is sustained
CHAPTER 1. INTRODUCTION
3
and increases as the user attempts to penetrate the object. If sufficiently strong, the force can prevent a user from penetrating a virtual wall in space. In comparison, a wearable device can only produce forces or vibrations that are grounded with respect to the user’s own body. For example, the Cybergrasp exoskeleton from Immersion Corp. [45] produces forces on a user’s fingertips by pulling on cables that are grounded with respect to the user’s arm. As the user touches a virtual object, the local forces applied to the fingertips can increase but there is nothing to stop the entire arm from penetrating a virtual wall. The price/performance tradeoff associated with with low-cost, portable haptic feedback, as compared to research-quality desktop devices, is an ongoing area of research. This dissertation addresses the effectiveness of low-cost, portable haptic feedback devices for training and for improving the ability of impaired individuals to perform manipulation tasks. The first part of this work investigates grounded and un-grounded haptic feedback in a training application. In this investigation, the grounded devices are inexpensive commercial force-feedback joysticks and the ungrounded devices are wearable displays adapted from commercial force-feedback computer mice. The training scenario is adapted from fire, police and military training exercises in which teams need to navigate and clear a dangerous building, identifying any victims that may be inside. The building is likely to be dark, smoky and noisy and the building-clearing operation must be done as quickly and efficiently as possible. In such situations, training procedures often include touch (e.g. tapping on the shoulder of a team mate) as part of the communication protocol [80]. In real environments, of course, personnel also receive tactile stimuli from all the objects that they come into contact with, intentionally or not. In recent years, there has been an increasing interest in virtual reality based training of procedures for such applications [75][83][107][70][21][29]. The motivation is that it is much easier and cheaper if at least some of the training can be conducted in a classroom using desktop computers instead of in an expensive outdoor training facility. It is now possible to obtain realistic visual and audio feedback in a virtual reality training system that can be run on a desktop PC. Due to the cost of desktop haptic
CHAPTER 1. INTRODUCTION
4
feedback devices, haptic feedback has not generally been a part of the solution. However, in other experiments, realtime haptic feedback has been shown to be effective in training procedures and in learning new motors skills (e.g. as related to a task or a job) [75] [5] [6] [10]. Several investigations have explored the use of haptic feedback in team-based training systems [84] [11] [40] [42]. The results of these studies suggest that the virtual training evironment does not have to be completely realistic in order for the benefits of training to carry over to realistic situations. However, most of the devices used thus far have been expensive, research-quality grounded force feedback devices. The work presented in Chapter 3 investigates whether low-cost commercial haptic devices can produce a significant improvement in the performance of users who are learning a procedure related to emergency building-clearing. The results of the experiments suggest that subjects do indeed learn procedures faster and with fewer errors using either grounded or ungrounded haptic feedback. In each case, accommodations must be made for the capabilities of the commercial feedback device. For example, in the case of force feedback joysticks, a smoothed and simplified version of contact forces was found to produce better results than attempting to replay virtual contact forces directly. Vibration feedback from devices attached to a user’s body produced nearly equivalent results, and this finding provided the motivation for the next sets of experiments investigating the use of vibrational haptic feedback for stroke and multiple sclerosis patients. For patients with neurological diseases, such as stroke and multiple sclerosis, it is common that tactile sensation is lost, diminished or distorted, which contributes to the difficulty they have in performing everyday tasks [39]. Many exercises are, at least initially, too difficult for these patients to perform without without the haptic feedback that they are missing. Simple repetitive exercises that can be performed are so tedious that the patients are not motivated to do them diligently. There is also a tendency to reduce the utilization of their impaired hands and shift tasks to the opposite hands [39]. The hypothesis behind the work in Chapters ?? and 5 is that if touch information on patients’ impaired hands can be collected and transmitted elsewhere to the patients, it might help them to accomplish more interesting tasks and
CHAPTER 1. INTRODUCTION
5
exercises sooner. Ultimately, the hope is that this might lead to faster rehabilitation, although to establish such a long-term effect is beyond the scope of this thesis. The work in Chapters ?? and 5 is motivated partly by previous findings that realtime haptic feedback can be useful in helping people to learn new motor skills [6] [10]. This result suggests that providing haptic feedback, initially to unimpaired areas that are capable of haptic sensation, might speed the rate at which patients learn, or relearn, to perform manual tasks. For stroke and multiple sclerosis, the sensory loss or reduction on upper extremities is often asymmetric, meaning that patients’ sensation may be missing on one hand and intact on the other. Knowing this fact, I developed a strategy of providing contact information from the fingers of patients’ impaired hands to corresponding fingers of their unimpaired hands. Using this approach, patients can“feel” the contact touch information from their impaired fingers on their their opposite fingers. This extra feedback provides a channel by which patients can achieved closed-loop control of the forces in the fingers of their impaired hands, allowing them to grasp objects more securely.
1.2
Thesis Overview
The entire dissertation consists of six chapters. The current chapter introduces the motivation behind the research, the thesis overview and the main contributions of this research.The second chapter reviews relevant research from the fields of virtual reality-based training and rehabilitation with haptic feedback. The third chapter presents a haptic feedback test bed for a virtual reality training system and experiments conducted with this test bed. A modified version of the Half-Life (v42/1.1.0.1) game engine was used to generate the virtual environment. In the first experiment, modified commercial force/vibration feedback joysticks were used as the haptic feedback rendering devices. In the second experiment, vibration feedback units were adapted from the inner mechanisms of commercial USB haptic mice. The results of both experiments showed that adding simple haptic feedback can significantly improve trainees’ performance in virtual reality training.
CHAPTER 1. INTRODUCTION
6
The fourth chapter introduces the research work on a haptic aid for stroke patients. The chapter begins with the results of a survey conducted at the Adaptive Learning Division for post-stroke recovery at Foothill College in Los Altos Hills, CA. The results of the survey indicated that many stroke patients have a sensory feedback loss on their impaired hand. Although most of patients would like to recover hand function rapidly, few are diligent about doing exercises. Major reasons for the patients’ lack of motivation to exercise more regularly are the tedium of the exercises and the slow apparent progress. In many patients, the impaired hand is always closed, making it difficult to do more interesting exercises and bimanual tasks. The chapter proceeds to describe the design and development of a novel device that would allow such patients to open their impaired hands using their opposite hands. The device went through several iterations, based on the results of tests with a small group of patients, to improve its adjustability, comfort level and efficacy. Finally, extra haptic feedback in the form of vibration was provided to the patients and an experiment was conducted to see if this haptic feedback could help them to gain a better grasping force control. The results of this pilot experiment were encouraging and lead to a more extensive and controlled set of experiments described in Chapter 5 on patients with multiple sclerosis. The fifth chapter addresses the use of force sensing and vibrational haptic feedback for patients with multiple sclerosis (MS). Usually MS patients have better motor capabilities than stroke patients so that they do not need to use the hand-opening device. We designed experiments to test the ability of haptic feedback to improve their control of grasping forces during a simple manipulation task. Special miniature force plates were designed to attached on the users’ impaired fingertips and force information was rendered to the corresponding fingernails on the opposite, unimpaired hands. The experiments showed that extra haptic feedback significantly improved patients’ ability to control their fingertip forces. We also found out that the less impaired subjects performed better with “event-cue” feedback (haptic feedback is provided only to alert subjects when their grasping forces stray outside of window of desirable forces), while the more impaired subjects performed better with an amplitude-based feedback (vibrations are directly correlated with the magnitudes of the grasp forces).
CHAPTER 1. INTRODUCTION
7
The sixth chapter summarizes the findings from the work considers future implications for work in portable haptic feedback aids for training and rehabilitation.
1.3
Contributions
The work described in this dissertation has provided several contributions to the field of haptic feedback for training and rehabilitation: • Experiments were conducted to investigate low-cost portable haptic feedback devices for the training of procedures in a virtual-reality environment. The results indicated that both force and vibration can improve the rate at which users learn new procedures and reduce the number of errors that they make. • A novel portable rehabilitation device was designed for stroke patients to allow them to open their impaired hands using their unimpaired hands. The device requires no external power and allows them to perform bimanual tasks earlier than might otherwise be possible. • For patients who have lost tactile sensation in one hand, a strategy was developed that uses small force sensors on the fingertips of the patient’s impaired hand in combination with haptic feedback provided elsewhere (e.g. on the patient’s opposite hand) to permit closed-loop control of grasping forces. As part of this work, a method was developed to map force magnitudes to the combined frequency and duty cycle of trains of pulses from a vibration stimulator. Experiments revealed that this mapping gave an approximately linear relationship between perceived magnitude and measured force, and was generally superior to a simple mapping of vibrator amplitude or frequency. • Experiments conducted with the force sensing and vibration feedback apparatus showed that patients could use it to achieve better control of grasp forces in a manipulation task. For multiple sclerosis patients, it was further discovered that there was a correlation between the patients’ impairment level and the type of feedback that produced best results.
Chapter 2 Background and Relevant Work 2.1
Background
Virtual reality-based training has become a popular topic for research and development due to its cost advantage and convenience compared with the conventional training in a full-scale real or simulated environment. Considerable effort has been devoted to the development of new hardware and software to support VR training. While these technologies provide more realistic immersive environments, the sense of touch is typically lacking [84]. In particular, for applications such as training for military and emergency personnel, there has been little use of haptics due at least in part to the requirements for mobility and a large workspace. A few specialized haptic feedback devices have been built that are suitable for VR training with mobile subjects, including [47], [8] and [86]. Interestingly, these devices are also suitable for applications such as rehabilitation, following stroke. However, these devices are expensive and lack portability. The work in this thesis is motivated by the idea that devices should ideally be worn on the body and should not be encumbering, so that subjects can wear them for extended periods of time without fatigue and without restricting their motions. In VR applications with haptics, attention has focused on increasing the realism of sensations. For example, systems may attempt to render collision with a stiff surface accurately, or at least so that the virtual wall imparts a subjective impression of 8
CHAPTER 2. BACKGROUND AND RELEVANT WORK
9
rigidity to the user [42] [84]. In the present work, I focus only on conveying unambiguously to the user whether he or she has contacted a wall, and with approximately what force. My hypothesis is that this information is more important than accurately rendering the contact dynamics and more important than being able to arrest the user’s motion. Under this assumption, small vibration and force feedback devices may be sufficient to impart a sensation of contact that a user can use to increase his performance in a task involving contacts. The same approach applies to the subsequent work on training and rehabilitation for stroke and MS patients. Real time haptic feedback has been shown to be valuable for motion training and new motor skill learning [6][10]. Mechanoreceptors all over the body are important for closed-loop control of motions, for example to maintain a consistent grasp force when handling objects [49]. These sensors also keep us continually informed about events such as the changes in force and the vibrations that accompany contacts. When the sensory feedback path is blocked or distorted, tasks such as tying a shoe or carrying a full glass of water become much more difficult. Many neurological diseases lead to sensory loss or distortion, such as stroke and multiple sclerosis [39]. The sensory loss creates a particular challenge for neurological rehabilitation, especially for upper extremity rehabilitation, since many exercises cannot be accomplished by a hand that has no sensory feedback. Various robotic and haptic devices have been designed to help patients to do rehabilitation exercises including those at the Palo Alto VA hospital. [12] [100] [67]. However, these systems are relatively bulky and expensive. In contrast, wearable haptic feedback devices could be used by patients in their homes. The following section will give a brief overview of systems and devices that are related to the work in this dissertation.
2.2
VR Systems for Mobile Persons
For emergency personnel training, VR-based systems have the advantages of increased safety, flexibility, repeatability and cost compared with traditional training [26]. In some cases, VR training may prove to be more effective than conventional training
CHAPTER 2. BACKGROUND AND RELEVANT WORK
10
in the physical word. For example, with a virtual reality system, trainees distributed in different places can be trained as a team at the same time. They are physically distributed, but virtually together.
Figure 2.1: Individual Combatant Simulators Parsons [75] presented a Fully Immersive Team Training (FITT) system that was designed for military or emergency personnel training using a virtual environment. The ultimate goal of the system design was to build a virtual environment that could do team training while the team members are geographically distributed. In the training environment, trainees were equipped with head-mounted display and six Ascension Flock of Birds trackers attached on different parts of their body to capture their motion (figure 2.1). Figure 2.1 shows two trainees participating in training from two different places. However, in the virtual reality training system, they were collaborating as a team. Preliminary experiments showed that FITT was a useful system for conducting research on distributed virtual environments for team training. Participants felt highly immersed in the team-training tasks and most displayed a great deal of motivation to succeed at their missions. However some limitations were also noted, such as a
CHAPTER 2. BACKGROUND AND RELEVANT WORK
11
Figure 2.2: Avatars as part of a virtual team narrow field of view and the absence of haptic feedback. Another advantage of virtual reality training is that team based exercises can be conducted with just a few live people, augmented by avatars simulated by computers. Traum [98] developed a teamwork model for virtual human agents, including a task model, a dialogue model, and aspects of an emotional evaluation model, all of which are integrated to allow complex team behavior, including negotiation and delegation. The model was tested by agents in a peacekeeping training scenario (figure 2.2). In other work involving manual skill acquisition, haptic feedback has been found useful for increasing the rate of learning [87][58]. For specific military and emergency personnel training tasks, it has also been shown that spatial skills learned in a virtual environment can transfer to real-world settings [105] [101]. However, due to the large scale and high mobility of the virtual reality system for military and emergency personnel training, it is often difficult to provide haptic feedback in such systems, although the disadvantage of having no haptic feedback has been noted [75]. To provide haptic feedback in a large virtual reality team training system is challenging and expensive. Nonetheless, several systems have been developed. Darken
CHAPTER 2. BACKGROUND AND RELEVANT WORK
12
Figure 2.3: Soldiers on Uniport and TreadPort [22] presented the work from his group on the three successive generations of locomotion devices, which were used in NPSNET [64]. The first generation (Uniport) was the first device built for lower body locomotion and exertion (figure 2.3). The Uniport operates in a similar manner to a cycle. Users pedal to simulate walking and running. With Uniport users can move forward, backward, turn left and right in the VR training system. Force feedback is provided when users are going up or down inclines, etc. The Treadport 2.3 was the second generation. It was based on a standard treadmill with user being monitored. A mechanical arm was attached to the user’s waist to provide force feedback to the user. This system is similar to Uniport; the main difference is that the Treadport allows users to walk and jog instead of pedaling. The Omni-Directional Treadmill 2.4 was the third generation. It was a revolutionary device for locomotion and providing force feedback in virtual environments, allowing users to walk in any direction on a treadmill. The force feedback was provided by the mechanical arm attached to the user’s waist. The active surface was 1.3 meters by 1.3 meters and the peak speed was 3 m/sec. Some practical deficiencies
CHAPTER 2. BACKGROUND AND RELEVANT WORK
13
were reported regarding the tracking system and control mechanism.
Figure 2.4: Soldier on the Omni-Directional Treadmill Other omni-directional treadmills include the Torus Treadmill developed by Iwata [48]. It employs twelve small treadmills connected side-by-side to form a large belt to allow arbitrary planar motion. The walkable area is 1 meter by 1 meter, and the maximum speed is 1.2 m/s. Motion control is achieved by foot tracking employing magnetic trackers. Christensen [16]. showed that the inertial force due to the acceleration of a locomotion interface is identified as a difference between virtual and real world locomotion. To counter the inertial force, an inertial-force feedback was implemented for the treadport. A force controller was designed for a mechanical tether to apply force feedback to the user. Simulations using simplied models showed that this controller should cancel the inertial force, however the actual implementation did not cancel the inertial force perfectly due to backlash of the tether-to-harness connection. User studies also showed that subjects preferred 80% of the inertial force feedback instead of 100% of the force feedback. A different solution to the problem of providing force feedback in an open VR environment is to attach devices directly to the user’s feet. The The Sarcos Biport
CHAPTER 2. BACKGROUND AND RELEVANT WORK
14
[42] is a exoskeleton style device that has three degrees of freedom actuated for each leg (figure 2.5). The picture on the left shows the Biport. When the user lifts a foot, the attached arm must follow with zero force to avoid dragging the foot. When the user steps to contact a surface, the arm must be servoed to present a rigid surface. This device has the advantage of being able to present different kinds of terrain, with varying stiffness.
Figure 2.5: The Sarcos Biport and The GaitMaster Iwata’s GaitMaster [47] is another device that can provide rich haptic feeling during navigation (figure 2.5). It comprises two 3-DOF parallel drive platforms. Each platform contains a passive spring-loaded jaw joint to allow some turning. The payload of each platform is 150kg. A 3-joint goniometer attaches a user’s foot to the platform to measure foot motion. This device also has the ability to render different kinds of terrain and stiffness. Other related devices include the Rutgers Ankle Haptic Interface presented by Boian [8] and the programmble haptic footplate system, Haptic Walker, designed by Henning Schmidt [86] (figure 2.6). Each of these devices has been tested in experiments concerning rehabilitation applications as well as virtual reality training.
CHAPTER 2. BACKGROUND AND RELEVANT WORK
15
Figure 2.6: The Haptic Walker Other related technologies have focused on providing haptic feedback and/or rehabilitation to the users upper limbs. Hirose’s HapticGEAR [40] is an example of a wearable force display. In order to provide freedom of motion in a large virtual space, the portable force display is grounded on the user’s back, as shown in figure 2.7. The force feedback is provided by the tension of wires controlled by electrical motors on the user’s back. This design is designed to reduce users’ fatigue and to have relatively little influence on the user’s motion and sight. Hashimoto’s SPIDAR-H [36] is a human scale haptic locomotion interface for virtual reality applications. It was developed from the smaller desktop version, SPIDAR [46]. Figure 2.8 shows an overview of a multi-projector display system on the left and a user wearing the SPIDAR-H haptic feedback device on the right. The design of the SPIDAR-H system is fairly simple. It has two rings to put on a user’s fingers on
CHAPTER 2. BACKGROUND AND RELEVANT WORK
16
Figure 2.7: The HapticGEAR
Figure 2.8: An overview of the multi-projector display D-vision and a user wearing the SPIDAR-H. Each hand has 4 cables to realize force feedback. each hand. Each finger has four cables that are pulled by motors mounted on the outer frame of the system. Encoders are used to determine the length of each cable, and the position of a user’s hand can be calculated by the length of the four cables. By using this system users can interact with virtual environment directly using their hands and, unlike the case with bulky devices, the users’ vision is not compromised. A device that is similar to SPIDAR-H was also presented in [11]. For neurological rehabilitation, other robotic and haptic applications have focused on the upper limbs. Casadio [13] introduced a haptic robot for multiple sclerosis patients. The system was originally designed for evaluation of motor learning and control, and for robot therapy. It consists of a two degree of freedom robot with
CHAPTER 2. BACKGROUND AND RELEVANT WORK
17
a handle attached to the end of the second link. The two DOF are both actuated by motors so that force fields can be produced within the working space. The wellknown force field adaptation paradigm [88] was used in the experiment. MS patients’ performance was compared with an equal number of age-matched control subjects. It was found that MS subjects displayed subtle coordination problems but did not otherwise significantly differ from the control subjects in their ability to adapt to the force field. Van der Loos [100] and Mahoney [67] presented several stroke rehabilitation robots developed at the Palo Alto Stroke Rehabilitation Center. Figure 2.9 shows the ARCMIME system and the MIME system. These systems involved relatively heavy machines that could lead the patient’s arm and hand motion.
Figure 2.9: The ARCMIME system and MIME system There are also systems that can provide haptic information for both upper and lower extremities. A Whole Body Kinesthetic Display Device for Virtual Reality Applications was described by Roston [84]. This design can provide a full six axis motion platform for each of the user’s feet. Kneeling boards are added to support rolling, kneeling and prone postures. Vertical feature presentation mechanisms allow the users to interact with realistic objects such as walls, windows, doors (figure 2.10). This is perhaps the most complex haptic system developed for virtual reality training.
CHAPTER 2. BACKGROUND AND RELEVANT WORK
18
Figure 2.10: A Whole Body Display
2.3
Wearable Systems
In contrast to using large systems that can impose grounded forces on a user’s arms or legs, other researchers have investigated small devices that can easily be worn on the user’s body. Vibrotactile feedback is the most common way of providing such feedback. It has been tested on different parts of human body (figure 2.11), such as the arms [72], waist [99], shoulders [97] and fingernails [2]. Information such as the occurrence of an event and the amplitude or direction of a quantity can be successfully delivered using combinations vibrotactile feedback. Haptic feedback can be useful for victims of stroke and other diseases. However, many stroke patients also need help in opening their impaired hand, which tends to remain closed with high muscle tone. Panasonic research [74] presented a stroke rehabilitation suit for elbow motion (figure 2.12). Elbow motion of the unimpaired
CHAPTER 2. BACKGROUND AND RELEVANT WORK
19
Figure 2.11: Vibrotactile display on arm, waist, shoulder and fingernail arm was tracked by bending sensors and artificial muscles were used as actuators to help the impaired arm to follow the motion of the unimpaired arm. This invention won the 2006 Times Magazine award for best inventions in a medical field. A similar strategy has been used for hand rehabilitation. Kawasaki [56] presented a hand motion assist robot (figure 2.13). The robot is an exoskeleton with 18 degrees of freedom and a self-motion control. The system continually tracks the motion of the unimpaired hand and allows the impaired hand of a patient to be driven by his or her healthy hand on the opposite side.
2.4
Conclusions
To create an authentic virtual reality environment for mobile subjects is a challenging problem. The solutions that can provide force feedback to a subject’s arms and legs tend to be complex, large and expensive. However, if we focus instead on providing haptic cues about events such as making contact, it becomes possible to use much lighter and simpler systems that can be worn on a subject’s body. There is some preliminary indication in the literature that such simple, body-worn systems can improve the experience and performance of subjects in a training exercise. We explore this question in more depth in the experiments in the next chapter. The idea of using inexpensive, compact, wearable haptic feedback devices is also
CHAPTER 2. BACKGROUND AND RELEVANT WORK
20
Figure 2.12: Panasonic Stroke Rehabilitation Suit for upper extremity appealing for patients suffering from stroke, multiple sclerosis or other diseases that greatly diminish haptic sensation. Considerable research has been devoted to robotic and haptic feedback systems aimed at rehabilitation. Leg and foot rehabilitation have received the most attention but some has been directed toward the upper limbs as well. Again, these systems tend to be relatively large and costly, so that patients need to travel to a center to use them. In contrast, compact, wearable haptic feedback systems could be used at home. However, a further complication with stroke patients is that providing haptic feedback is often not sufficient; mechanical assistance is also required to open the impaired hand. In this case, a manually powered solution (powered by the patient, perhaps using the unimpaired opposite limb) has the advantage of being relatively light, simple and easy to control as compared to solutions that require actuators, amplifiers and a large power supply.
CHAPTER 2. BACKGROUND AND RELEVANT WORK
Figure 2.13: The Hand Motion Assist Robot
21
Chapter 3 Haptic Feedback for Virtual Reality Training As explained in Chapter 2, there has been an increasing interest in virtual reality based training of procedures for military, police and emergency personnel [75][83][107] [70][21][29]. An encouraging result of early studies [4][14] is that virtual reality environments need not be entirely realistic in order to provide useful training that carries over to real situations. Most virtual reality-based training has focused on visual and auditory feedback. Haptic feedback has been explored in a few cases [61][15][91] with promising results. High-end ”immersive” virtual reality systems may include wearable head-mounted displays [102] or ”cave” video projection systems[19], three-dimensional motion tracking [43] and perhaps treadmills [90][22] or harnesses for imparting resistance to the motion of the subject [16]. These systems are capable of kinesthetic as well as visual and auditory feedback [42][26][76]. Although kinesthetic feedback has the potential to provide a very high level of realism, the costs and mechanical complexity have made the technology prohibitive for wide scale use. Furthermore, such systems can usually only train either one or a few people at a time. This is particularly a drawback when large groups of people should be trained together. Consequently there has been an interest in low-cost virtual reality environments, such as those found in multi-user video games for desktop computers. The hope is that with steady improvements in desktop 22
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
23
technology, these low-cost VR trainers will be responsive and realistic enough to help subjects learn useful procedures. In these virtual reality training applications, subjects view the scenes using either computer monitors or inexpensive head-mounted displays and impart motions and commands using joysticks, keyboards and other commercial gaming devices. The selection of USB-compatible gaming devices with haptic feedback is steadily growing, which leads to the following questions: • What roles can haptic feedback play in low-cost virtual reality training for military and emergency personnel? • How can performance or learning rates be improved with haptic feedback? To shed light on these questions I undertook experiments involving the addition of haptic feedback in a low-cost virtual reality training scenario. The focus of the work, which was conducted in collaboration with Immersion Corp. in San Jose and funded by the Office of Naval Research, was training for building-clearing operations as practiced by military personnel in close-quarter combat and by emergency personnel to evacuate hostages or earthquake victims. The challenges in such environments often include poor visibility, distracting noises (e.g. explosions), and a severe time pressure for planning and executing procedures. Haptic feedback provides a useful additional channel of information and communication. For example, in some procedures personnel are trained to tap the shoulder of a team-mate as part of the communication protocol for Close Quarter Battles (CQB) [80]. In real environments, of course, personnel also receive tactile stimuli from all the objects that they come into contact with, intentionally or not.
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
3.1
24
Experiment One
The aim of the first experiment was to investigate the effects of haptic feedback on a subject’s ability to remember and accurately execute procedures while negotiating a virtual environment. A commercial graphics engine, Half Life, (v42/1.1.0.1) was used for the experiment. Users were asked to negotiate a simulated environment using standard video game joysticks, with or without force and vibration feedback. The question addressed was whether haptic feedback could improve users’ performance or increase their learning rate.
3.1.1
Experiment One Setup
Figure 3.1: Screenshot one of half life environment as used in experiment one.
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
25
Our experiments were conducted using a dual-processor Windows desktop running a modified version of the Half- Life (v42/1.1.0.1) game engine which is capable of generating kinesthetic and tactile feedback as a result of players’ actions in the virtual environment. Two screen shots of the running application are shown in Figure 3.1 and Figure 3.2. In addition to producing cues for haptic feedback, the modification logged the player’s position, velocity, collision state and “clip fraction,” a variable defined by the difference between the desired and achieved, or clipped, velocity when users run into an obstacle: (Vdes − Vclip )/Vmax · Vdes and Vclip are two dimensional vectors of forward and lateral velocity components and Vmax = 320 units/16ms is the maximum speed a player can obtain. For kinesthetic effects, the direction of the effect was updated based on player orientation every 16ms. Unfortunately, like many game engines, Half-Life does not produce more detailed collision information, such as penetration depth or geometric details. Because of this limitation, the magnitudes of the haptic feedback effects were made proportional to the clip fraction at the initial collision. The method for computing the force is shown in Figure 3.3. The interaction force is computed based on the difference between the user’s desired velocity and actual velocity after contact with the wall. This force value is then held constant, or “latched,” until the user breaks contact with the wall. The resulting contact force is a simplified approximation to the contact force, without the dynamic variations that a real interaction force would have. In addition to collision feedback, two special textures were implemented to allow the system to display different tactile effects during player walkover. This addition allowed effects to be played through different feedback devices depending on the texture. Thus, a texture associated with low-lying obstacles could be routed through different devices than a texture applied to shoulder-height obstacles. To achieve haptic effects, we modified commercial force feedback joysticks (Saitek Cyborg 3D Force) and vibration devices (Saitek Cyborg 3D Rumble Force) with USB drivers from Immersion Corporation. Preliminary tests were first conducted to determine under what conditions haptic feedback could give meaningful cues. In many cases, the initial results were disappointing. Experienced video game players performed so well using visual cues that
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
26
Figure 3.2: Screenshot two of half life environment as used in experiment one. haptic feedback was of little consequence. However, it gradually became clear that under certain conditions players could learn an environment faster and make fewer errors with appropriate haptic cues. Accordingly, the following scenario, inspired by Close Quarters Battle (CQB) training manuals, was developed for test in experiment one.
3.1.2
Scenario
Imagine a training session in a simulated environment for rescue missions. A trainee has to rescue hostages, or perhaps survivors of an explosion, from inside of a dark and dangerous building. The building must be cleared and the hostages must be recovered quickly. However, it is important to check each room for safety before entering. To
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
27
Computing collision force from Vdesired before the collision and Vactual after the collision
Y
F = - k(dV) #$#%#&"
Vdesired
Vactual #$#%#&"
Vdesired dV
Wall
X Figure 3.3: Diagram of clipping effect used to approximate the collision force experienced when a user contacts a wall. indicate that a room has been checked the user must briefly stop against a wall at either side of the entryway [41]. The task is considered to be complete when users complete a sweep of the building and proceed through an exit at the far end from the starting point (Figure 3.4). When haptic feedback (joystick force or vibration) is on, contacts with any obstacles are registered, including the walls just outside the room to be cleared. The measured variables included the total time to complete the mission and the number of failures to properly check rooms before entering.
!"
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
28
Figure 3.4: The user must stop against the wall prior to entering a new room.
3.1.3
Protocol
A set of twelve diverse subjects, (eight male and four female, with ages ranging from 20 to 30 years), six having prior experience in 3D gaming and six having little or no experience, were chosen for the experiment. Three feedback modes were selected for this experiment. A: No Haptic Feedback B: Vibrotactile Feedback using a commercial vibration effects joystick C: Force Feedback using a commercial force feedback joystick.
Subject1 Subject2 Subject3 Subject4 Subject5 Subject6
Run1 CBA ABC BAC ACB BCA CAB
Run2 CBA ABC BAC ACB BCA CAB
Run3 CBA ABC BAC ACB BCA CAB
Run4 CBA ABC BAC ACB BCA CAB
Table 3.1: Pattern of testing for subject groups (6 experienced and 6 inexperienced) In case B, a vibration of approximately 30Hz and a peak amplitude of approximately 2G (for a typical grip on the joystick handle) was produced when subjects came into contact with an object (e.g. a wall) in the virtual environment. In case C, a force of up to 6 Newton was produced, depending on the severity of the contact, based on the clip fraction as discussed in the experiment setup section.
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
29
Each subject was assigned a sequence of haptic feedback modes (A, B, or C), such that all six combinations of A, B and C were covered. This was done for both groups (experienced and inexperienced). The order of presentation was randomized and balanced. (Table 3.1) Each subject carried out twelve trials in total, i.e. four runs with three trials each (Table 3.1). There were twelve different maps of identical complexity level, one for each trial. The choice of maps was completely random and was decided before the test. One example map is shown in Figure 3.5
Figure 3.5: Map of a typical building layout in experiment one
3.1.4
Metrics
The primary metric was the subjects’ ability to complete the mission without failing to register a safety check by touching a wall just outside a room before entering it. The number of hostages rescued was not a reliable statistic because it relied mainly on the user’s ability to see hostages, despite having a narrow field of view and a darkened environment.
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
3.1.5
30
Data Analysis and Results
Inexperienced Subject1 Average Number of Subject2 incorrect Entries for Subject3 each subject and each Subject4 feedback mode Subject5 Subject6
A 1.75 1.00 1.50 3.00 3.00 2.30
B 0.25 0.75 1.75 0.25 2.30 0.30
C 1.00 0.50 0.50 0.75 2.30 0.30
Table 3.2: Average numbers of errors (failures to check rooms properly before entering) for inexperienced video game players. Four trials per subject per condition: A = no haptic feedback, B = joystick vibration feedback, C = joystick force feedback.
Experienced Subject1 Average Number of Subject2 incorrect Entries for Subject3 each subject and each Subject4 feedback mode Subject5 Subject6
A 0.25 1.25 5.00 3.50 0.25 1.25
B 0.25 0.50 2.25 1.25 0.00 0.00
C 0.00 0.50 2.50 1.25 0.00 1.00
Table 3.3: Average numbers of errors (failures to check rooms properly before entering) for experienced video game players. Four trials per subject per condition: A = no haptic feedback, B = joystick vibration feedback, C = joystick force feedback. Because early testing revealed significant differences in the strategies used by experienced and inexperienced video game players, the results were initially divided into two pools, as listed in Table 3.2 and Table 3.3.
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
31
Experiment II Results
Figure 3.6: Box plots showing the error statistics for experienced and inexperienced subjects in experiment one.
!"
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
32
As seen in Figure 3.6, there is a significant difference in the number of errors made when using either vibration or force feedback as compared to the no-haptics case (F = 4.74, P = 0.0155 in an ANOVA test). Considered separately, the case of vibration has a significantly different mean (P = 0.003) as compared to no-haptics, as does the case of force feedback versus no-haptics (P = 0.001). Because early testing revealed differences in the strategies used by experienced and inexperienced video game players, the results were divided into pools of six experienced and six inexperienced users. The averages for each pool are labeled in Figure 3.6
3.1.6
Learning Rates
The errors produced by experienced and inexperienced subjects are plotted (Figure 3.7) as a function of the trial number (each subject had four trials with each condition: force feedback, vibration feedback, no feedback). The results show that despite substantial subject-to-subject variability, there is a slight reduction in the average number of errors when progressing from trial 1 to 4. Conducting a paired-T test for all subjects between trial 4 and trial 1 reveals that the probability of a statistically significant difference in the means is 98.5% (P=0.015) for condition A (no haptics), 95.8% (P = 0.042) for condition B (vibration feedback) and 76.4% (P=0.236) for condition C (force feedback).
3.1.7
Experiment One Conclusion
As Figure 3.6 indicates, there is a clear reduction in the average number of errors made when experiment includes either vibration or force feedback. Also, there appears to be no significant difference between force or vibrotactile feedback when using this performance metric. From Figure 3.7, there is some evidence of learning over the four trials; however the learning is actually more evident without haptics than with them. One way to interpret these results is that the addition of haptics immediately improved performance to an extent that little further improvement was obtained over four trials.
!"#$%&'$()*+$,-.),*
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
33
Figure 3.7: Error statistics for subjects as a function of trial number. There is a slight reduction in the average number of errors between trials 1 and 4. (Recall that the presentation order was randomized to guard against bias.) Anecdotally, all subjects reported a greater sense of immersion with haptic feedback. The case with force feedback was slightly preferred. However, it should be noted that this result was only obtained after considerable experimentation and modification of the force computation in preliminary tests. Initially, the limitations of the video game engine (which provides no information about the details of collisions with objects) and a commercial force feedback joystick produced results that users found more distracting than helpful. Useful force feedback was obtained only after implementing an algorithm in which the initial force was made proportional to the user’s
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
34
velocity in the direction normal to the collision surface (e.g. a wall) and then latched at the value until the user departed from the surface. The resulting force, while not exactly realistic, is smooth and provides useful information about the direction and magnitude of a collision. More generally, the results of this experiment lead us to believe that haptic feedback in a virtual environment can reduce the number of errors made by subjects executing a critical procedure modeled after procedures used in building-clearing.
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
3.2 3.2.1
35
Experiment Two Experiment Two Setup
Figure 3.8: Experiment setup side view: two vibration tactors were attached to the head of the subject and two vibration tactors were attached to the legs of the subject As in experiment one, the Half Life video game engine was used to develop the virtual environment. The vibration feedback units were adapted from the inner mechanisms of commercial USB haptic mice (Figure 3.10, 3.11). They were attached to Velcro straps that could be fastened to various parts of the users’ bodies. After some experimentation, the best results were obtained using four vibration devices, with two attached to a user’s head (for collisions with high obstacles) and two attached to a user’s lower legs (for low collisions) (Figures 3.8, 3.9). Empirically, best results were obtained with vibrations of approximately 10Hz and a 6.4G peak amplitude at the head and of 100Hz and a 5.7G peak amplitude at the legs. Consequently, the
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
36
Figure 3.9: Experiment setup front view: two vibration tactors were attached to the head of the subject and two vibration tactors were attached to the legs of the subject. vibrations were distinct in terms of frequency as well as location. Although Half Life engine does not provide detailed information about collisions, it does provide information about the texture of the surface that a subject is traversing. Two special textures, for high and low obstacles, were mapped onto the floor just in front of each respective obstacle. The texture regions were made narrow enough so that a subject’s avatar would be found standing over them only if a collision occurred (Figure 3.12). By checking the texture regions we were able to determine whether a collision occurred and which type of obstacle was hit. This information triggered the corresponding vibration device.
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
Figure 3.10: Top view of the vibration tactor
Figure 3.11: Side view of the vibration tactor
37
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
3.2.2
38
Scenario
The focus of experiment two was to evaluate the effects of distributed vibration feedback on a user’s body during a virtual reality training exercise. The task given to users was to guide an avatar through a dark, cluttered and potentially hazardous environment – perhaps a building or tunnel in the aftermath of an explosion. In such applications, it is often valuable for personnel to retain an accurate memory of the condition and obstacles for future reference. In this scenario, haptic feedback stands as a substitute for the typically undesired sensations that contact with real objects would normally produce. In our virtual environment, the user must go through a dim corridor. There are two main kinds of obstacles: low ones that must be jumped or stepped over and high ones that must be ducked under to prevent head injuries (Figure 3.12). The total number of obstacles in the corridor is 15, and they are in random order. A dim red light indicates the direction of the exit.
Figure 3.12: Layout of the corridor in experiment two. A standard video game mouse and keyboard interface was provided for input. The questions being addressed included whether haptic feedback allows users to complete the task in less time and whether they can better remember the details of the environment that they have negotiated.
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
3.2.3
39
Protocol
A diverse set of eight subjects (5 male and 3 female, with ages ranging from 20 to 30 years and with varying degrees of video game experience) were chosen for the experiment. Four variations on the corridor were used with sequences of 15 obstacles shown in Table 3.4.
Corridor Corridor Corridor Corridor
A B C D
Sequence of obstacles (High, Low) HLLHHLHLLLHLHHL LHHLHLLLHLHHLLH LHHLLHLHHHLHLLH HLLHLHHHLHLLHHL
B is the reversed sequence of A C is the conjugate sequence of A D is the reversed conjugate sequence of A
Table 3.4: Sequences of obstacles encountered in the test corridors for Experiment 2
Subject1 Subject2 Subject3 Subject4 Subject5 Subject6 Subject7 Subject8
Speed Trial One A(Haptics) A(No Hap) B(Haptics) B(No Hap) A(Haptics) A(No Hap) B(Haptics) B(No Hap)
Tests Trial Two C(No Hap) C(Haptics) D(No Hap) D(Haptics) D(No Hap) D(Haptics) C(No Hap) C(Haptics)
Memory Tests Trial Three Trial Four B(Haptics) D(No Hap) B(No Hap) D(Haptics) A(Haptics) C(No Hap) A(No Hap) C(Haptics) B(Haptics) C(No Hap) B(No Hap) C(Haptics) A(Haptics) D(No Hap) A(No Hap) D(Haptics)
Table 3.5: Sequences of corridors and trials, with or without haptic feedback, for eight subjects Each subject carried out 8 trials in total, two trials in each of the four corridors. The order of presentation was varied and balanced among subjects as shown in Table 3.5. For the first four trials, subjects were asked to go as fast as possible. For the second four trials, the subjects were informed that they had one minute (ample time to negotiate the corridor) and that they would be asked afterward to try to recall the sequence of obstacles they had encountered along the way.
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
3.2.4
40
Metrics
The measured variables included the total time required and, for memory trials, the number of obstacles of each type (high or low) correctly remembered after completing a trial.
3.2.5
!"#$%&'$()*+$,-.),*
Data Analysis and Results
Figure 3.13: Obstacles (total reported) and number of obstacles correctly recorded, with or without haptic feedback. Figure 3.13 shows the total numbers of obstacles that users reported and the numbers of obstacles correctly identified as high or low in the sequence. The box plots show the median and upper and lower quartiles, as well as the maximum and minimum values found across all subjects. The average numbers of obstacles are also shown for each box. The maximum possible number of correctly identified obstacles is 15. Interestingly,
41 !"#$%&'$()*+$,-.),*
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
the presence of haptic feedback caused users both to report more objects and to recall more objects correctly. Conducting a paired T-test between the case of haptics versus no-haptics for the number of obstacles correctly identified results in a probability of 98% for a statistically significant difference in the averages ( P = 0.021).
Figure 3.14: Histogram of the relative time required to complete a trial with haptics versus without haptics for the two speed trials. There were also significant differences in the amount of time that people took to negotiate the corridor with and without haptics. Recall that for the first two trials, speed was emphasized and that for the second two trials memory was emphasized, rather than speed. Nonetheless, most users completed the trials more rapidly with haptic feedback. Figures 3.15 and 3.14 show histograms of the relative amount of time required by each subject to complete a speed trial with haptics versus without: R = Thaptic /Tnohaptic . For the speed trials, we see that four of the eight subjects required between 90% and 100% as long to complete the task with haptics as without, and three required less than 90% as long, and one took slightly longer with haptics. Performing a paired T-test on the average amounts of time for the speed trials, with and without haptics, results in a 94% (P = 0.06) confidence in a different average time. Although speed was not a goal in the memory trials, most subjects again completed the task
!"#$%&'$()*+$,-.),* CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
42
Figure 3.15: Histogram of the relative time required to complete a trial with haptics versus without haptics for the two memory trials. Although speed was not a goal in these trials, all but two subjects completed the task faster when haptic feedback was present. faster when haptic feedback was present.
3.2.6
Experiment Two Conclusions
The provision of vibrational haptic feedback to users’ heads and lower legs helped users to identify high and low obstacles more quickly and more accurately than in the case when haptic feedback was absent. The numbers of obstacles correctly identified and the total task times were superior when haptic feedback was present. While watching the users, it was clear that they recovered from collisions more rapidly with haptic feedback. Although there was some spatial information associated with different feedback devices located at the head and lower limbs, we hypothesize that this is still primarily an example of ”temporal” or ”event” feedback, which other investigators have also found to be effectively conveyed via haptic feedback. Thus, users detected “head events” or “foot events” more quickly and more memorably with haptic feedback.
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
43
A comment should also be made about the approaches taken by experienced versus novice video game players. In fact, original scenarios developed during some preliminary tests proved too easy for accomplished gamers. They proceeded extremely rapidly through the corridor and made virtually no mistakes with utilizing only visual feedback. By darkening the environment, the task became sufficiently challenging so that all users benefited from the extra feedback that haptics provided.
CHAPTER 3. HAPTIC FEEDBACK FOR VIRTUAL REALITY TRAINING
3.3
44
Discussions and Conclusions
The results from the two experiments lead us to believe that haptic feedback can play an important role in low-cost VR training of military and emergency personnel. The experiments made use of a modified commercial videogame engine, haptic feedback devices and USB device drivers. Experiment one showed that subjects who utilized either vibration or force feedback made significantly fewer errors, implying higher levels of immersion. In preliminary tests, a force feedback joystick performed poorly in comparison to vibration feedback. After modifications to the collision detection algorithm, a simplified form of force feedback provided results equal to those obtained with vibration feedback and was preferred by most subjects. Experiment two showed an improvement in speed of performance and in accuracy of recall, when negotiating a dark, cluttered environment using small vibration feedback devices mounted to the user’s head and legs. The overall conclusion from these experiments is that haptic feedback can help subjects to learn procedures in a virtual environment and make fewer errors while executing procedures. Moreover, the feedback does not need to provide a realistic display of interaction forces. Low-fidelity rendering of contact forces may be more distracting than helpful. When using inexpensive, mass market feedback devices, a simplified approximation to the contact forces gives better results.
Chapter 4 Enhancing Stroke Rehabilitation 4.1
Introduction
In the previous chapter it was found that adding simple haptic feedback cues could help people navigating in a VR environment by approximating some of the cues that they would normally receive in a corresponding scenario. Another class of applications in which normal haptic feedback is missing includes pathologies such as stroke and multiple sclerosis, which often result in distorted or greatly reduced haptic and proprioceptive sensation. In these applications, haptic feedback applied to an unimpaired region of the body could potentially substitute for missing haptic cues. Stroke is a leading cause of disability among adults in the United States [33]. Many stroke patients with upper arm hemiparesis exhibit an adaptive trend of underuse of their affected side hand and overuse of their less-affected side [54]. This trend can delay recovery of the affected side. In recognition of this trend, various devices and methods have been developed to help patients increase use of the affected side. For example, Van der Loos [100] and Mahoney [67] presented several stroke rehabilitation robots developed at the VA Palo Alto Rehabilitation R&D Center. Johnson and Van der Loos developed a Robot-assisted driving environment, Driver’s SEAT [54], and Taub invented Constraint-Induced Movement Therapy[95, 93, 94]. An important consideration in such efforts is motivation: the instrumented driver’s steering wheel [54] is particularly effective because stroke patients are motivated to recover the ability 45
CHAPTER 4. ENHANCING STROKE REHABILITATION
46
to drive. For stroke patients, it is common that haptic information is greatly diminished or absent on the impaired side. However, it is well established that sensory feedback is critical for people to perform basic manipulations of objects [6, 10] and to control grasp forces [50, 51, 52, 53]. Consequently, it is not surprising that a lack of sensory feedback is one of the obstacles that deter stroke patients from using their impaired hands [55, 33, 34, 31]. This observation motivates an approach in which touch information is collected via electronic sensors at the impaired hand and used to provide haptic feedback elsewhere on the body. With the provision of such haptic feedback, patients could be encouraged to use their impaired hands for more activities and exercises. However, in the case of stroke patients, there is an additional problem that must be overcome: the impaired hand often cannot be opened. Part of the problem is that in the human hand, the muscles responsible for closing the hand are stronger than the muscles that open the hand. During the rehabilitation process, the larger muscles usually recover faster, but they also develop spasticity. Kamper [25] argued that persistent and inappropriate flexor activation plays a role in limiting voluntary finger extension. In other words, the imbalance between the weak extensors and spastic flexors causes the hand opening problem. This inability also prevents these patients from doing manipulation exercises. Therefore, before haptic feedback can be used to help such patients, they require a system that allows them to open and close the impaired hand at will. Several devices have been developed that help patients to open their impaired hands using different approaches, such as motors [56], preloaded springs (SaeboFlex [23]), electrical stimulus (Ness H200 [79]), air pump (The KMI Hand Mentor [59]) and artificial muscles (Panasonic suite [74]). However, these systems lack the combination of manual user control, portability, and provisions for adding haptic feedback found in the system described in this chapter. The remainder of this chapter has three parts. First, the chapter describes a survey conducted with stroke patients in order to obtain direct information about hand functionality and attitudes regarding exercises to promote rehabilitation. Next, the chapter introduces a new portable device that helps stroke patients to open their
CHAPTER 4. ENHANCING STROKE REHABILITATION
47
impaired hands using power from their unimpaired hands. Finally, it discusses a pilot experiment designed to see whether the provision of extra haptic feedback can affect how stroke subjects control forces on their impaired hands and whether different types of haptic feedback have different effect on subjects’ performance. The encouraging results of this pilot study lead to the more detailed experiments reported in the next chapter on multiple sclerosis patients.
4.2
A Survey of Stroke Patients
To get information about haptic sensory feedback and the real problems encountered by stroke patients, a survey was conducted in collaboration with the Adaptive Learning Division for post-stroke recovery at Foothill College1 . Nineteen subjects participated in the survey. The period of time following the stroke ranged from one to twenty years. The results and insights from the survey are discussed in this section. Particular thanks are due to therapist Deborah Kenney for her assistance in arranging for the tests and surveys described in this chapter.
4.2.1
Survey Questions and Results
First, subjects were asked to report whether they had any problems with their hands or arms after the stroke. Among all 19 subjects, 18 reported that they had problems; one reported “No.” Second, subjects were asked to report the state of sensory feedback in their impaired hands. Out of the 18 subjects who reported problems, six said they had normal sensation, five reported no sensation at all and seven reported a distorted sensation. Among subjects who reported that they had sensation after a stroke, some said that they felt their sensation was okay, but they were not sure if it was normal. Third, subjects were asked to report how many hours they usually spent doing rehabilitation exercises each day. The histogram in figure 4.1 shows the trend in this sample group. 1
http://www.foothill.edu/al/index.php
CHAPTER 4. ENHANCING STROKE REHABILITATION
48
Figure 4.1: Number of hours subjects spent on rehabiliation exercises daily The plot peaks at approximately 1 hour. Among all subjects, 68.4% do rehabilitation exercises less than or equal to one hour a day and 15.8% do no rehabilitation exercises. Only one out of 19 subjects exercised more than three hours a day. Fourth, since many stroke patients are not very proactive about hand rehabilitation, subjects were asked to report how important it is to recover their hand function. Three choices were provided: “Important,” “not that important,” and “I can do most of the jobs with my unimpaired hand.” Subjects were allowed to choose more than one option. The results showed that 89.5% of the subjects thought it is important to recover their hand function, 10.5% thought it is not that important and 15.8% thought that they could do most of the jobs with their unimpaired hands.
4.2.2
Discussion
From the data acquired in this survey, we can see a gap between stroke patients’ desire and effort spent on rehabilitation exercises. The survey shows that most subjects had hand problems and more than two thirds have lost, distorted or reduced haptic feedback. More importantly, although most of the patients considered it very important to have their hand function recovered, more than two thirds did relatively little exercise. This lack of motivation is also observed and discussed by many other researchers [65, 66, 44, 96, 57]. After observing stroke patients’ rehabilitation sessions at the Stanford University
CHAPTER 4. ENHANCING STROKE REHABILITATION
49
Hospital and in Foothill’s REACH program for about a month and conducting more detailed interviews with therapists and stroke patients, it was found that several factors were responsible for patients having a lack of motivation to do more exercises. First, the improvement over time for stroke patients is very slow. It can take months to make a small amount of progress. Second, most rehabilitation exercises consist of repeating simple motions, which is boring. Patients’ interest in doing these exercises reduces rapidly with time. Third, without haptic feedback and the ability to open their impaired hand, it is very difficult for the patients to do exercises that are interesting and practical (for example, that correspond to tasks such as carrying a tray). Fourth, after leaving the hospital, many patients cannot get effective rehabilitation services since many good rehabilitation devices are only available in hospitals. Although the first factor, low improvement rate, is hard to change, the other three factors can be addressed. A portable and low-cost system that allows patients to open their impaired arms and provides haptic feedback could allow them to perform more interesting and task-oriented exercises at home as well as at the rehabilitation center. In the next section, a new hand-opening system is described, along with the objectives that govern its design.
4.3
Design and Development of a Portable HandOpening Device
4.3.1
Design Goals
The hand-opening system depicted in figures 4.2 and 4.3 was developed through several prototypes with informal user tests at each iteration. The main design goals for the system are: 1. The system should give patients control, and a sense of being in control, over the process of opening and closing their impaired hands. 2. The system should be light and portable. Mobility is highly desirable. Most rehabilitation devices with motors, transmissions, air pumps, etc. are heavy,
CHAPTER 4. ENHANCING STROKE REHABILITATION
50
making it difficult to take them from room to room as patients move about. 3. A corollary to the above requirement is that the system should ideally not require an external power source. If the device has to be plugged into the wall, the power cord will greatly limit mobility. If it uses batteries, they will add to its weight (and will need to be replaced or recharged regularly). An additional advantage to using no external power is that it is much easier to ensure safety in the event of any component failure or malfunction. 4. The system should be easy and comfortable to wear. It can be very difficult for stroke patients to don and doff items on the impaired hand, given its spasticity and reduced control. It should also be easy for patients with various hand sizes to adjust the device so that it becomes comfortable. 5. The device should be simple, robust and not expensive to manufacture so that it can applied as widely as possible. Some additional desirable features were identified during the design iterations and incorporated into the final design. For example, a hydraulic master/slave system, with its combination of stiffness and viscous damping, provided patients with a feeling of security in controlling the impaired hand. A design that keeps the inside of the impaired hand free to grasp objects was also found to be desirable.
4.3.2
Design Description
The hand-opening device consists of two subsystems. The first of these is designed to be worn on the patient’s impaired hand; the second is worn on the unimpaired hand. With this device, patient’s two hands open and close at the same time. When the patient try to open his/her impaired hand, the opening force from the unimpaired hand will be transferred to the impaired side through a hydraulic and cable-drive system to help the unimpaired hand open. Figures 4.2 and 4.3 show the subsystems on each hand in the closed and opened configurations, respectively. Details of the designs are provided in figures 4.5 and the accompanying text.
CHAPTER 4. ENHANCING STROKE REHABILITATION
51
Figure 4.2: Impaired (left) and unimpaired (right) hands wearing their respective subsystems of the hand-opening device, shown in the closed (flexed) configuration.
CHAPTER 4. ENHANCING STROKE REHABILITATION
52
Figure 4.3: Impaired (left) and unimpaired (right) hands with their respective subsystems of the hand-opening device, shown in the opened configuration.
CHAPTER 4. ENHANCING STROKE REHABILITATION
53
Most of the components of the hand-openig in this device were made using the Shape Deposition Manufacturing (SDM) rapid prototyping process at Stanford. The basic cycle of SDM is shown in Figure 4.4 and consists of adding and shaping materials. SDM parts can consist of multiple materials and can include embedded components [103, 7]. In the present case, the materials are hard (Shore 90 D) urethanes and shaping is done by CNC machining. With SDM technology, it is possible to make complex 3D parts in a short period of time. This capability is useful for fast design iterations but in the present application an equally important advantage is that parametric 3D CAD models are used for the components. The parametric models can easily be scaled over a range of sizes to create collections of parts that accommodate various sizes of fingers. Device subsystem for the imparied hand Figure 4.5 shows the main components of the subsystem worn on the impaired hand. The components include finger holders, a cable guidance system, a slave cylinder and piston support, and a skin protector and wrist brace. Finger Holders The finger holders are shown in figures 4.7 - 4.9. Although all finger holders have the same general design, they are created at slightly different sizes to fit different sizes of fingers (within and across subjects). The main adjustable dimensions are shown in figure 4.7. The main components of the finger holders are identified in figure 4.7. The thimble, shown in green, transmits force to the fingertip. The tail section, shown in brown, includes a sector pulley and cable guide. The sector pulley increases the torque about the metacarpophalangeal joint and prevents accidental buckling, as seen in figure 4.8. The cable adjustment mechanism, shown in blue with a yellow slider, has a screw, shown in figure 4.9, to provide 7mm of adjustment of the cable termination location for accommodating different lengths of finger phalanges. Skin Protector and Wrist Brace Since the device needs to be mounted firmly on the patient’s hand and will experience forces of several N, a wrist brace and fabric sleeve are needed for comfort
CHAPTER 4. ENHANCING STROKE REHABILITATION
54
Figure 4.4: Shape Deposition Manufacturing cycle involving material additional removal and part embedding
CHAPTER 4. ENHANCING STROKE REHABILITATION
Figure 4.5: Top view of the hand-opening device for the impaired hand
Figure 4.6: Top view of hand-opening device when the hand is open
55
CHAPTER 4. ENHANCING STROKE REHABILITATION
56
Figure 4.7: Finger Holder parametric CAD model. The marked dimensions are in millimeters and are adjusted to accommodate different finger sizes. and support, as shown in figure 4.10. (Other devices such as the SaeboFlex [23] have a brace and sleeve for the similar reasons.) Without this wrist brace, many stroke patients have a negative angle between the palm and the forearm, which prevents them from doing many grasping exercises. Back Support and Cable Guides The main function of the back support is to guide the cables that connect the fingers to the cylinder and piston assembly. The support is attached with velcro straps, as shown in figure 4.11. The angles of the finger cable guides are adjusted for different hand sizes and the thumb guide can be positioned along a curved track. Cylinder and Piston Guide The cylinder and piston guide are shown in figure 4.12. The cylinder has approximately 40mm stroke and is made of plastic, with a stainless steel piston and rod so that water can be used as the hydraulic fluid.
CHAPTER 4. ENHANCING STROKE REHABILITATION
57
Figure 4.8: The tail section of the finger holder (left) prevents buckling of the metacarpophalangeal joint (right). Subsystem for the Unimpaired Side Figure 4.13 shows the subsystem worn by the unimpaired side, which is used to provide power to the impaired hand. The subsystem consists of three main parts: a skin-protecting sleeve, a master cylinder assembly and finger holders. The master cylinder has a similar piston guide as seen in the subsystem for the impaired hand. Because the muscles responsible for opening the hand are usually not that strong, a compression spring has been added around the piston rod to make it easier to open the hand against the force of the impaired hand. In this case, the finger holders are simpler than those for the impaired hand. The goal is to make it as easy as possible to put them on and adjust them and there is less concern about providing stability than there is for the impaired hand. Figure 4.14 shows two images of finger holders of different sizes. As with the finger holders for the impaired hand, a parametric CAD model makes it possible to quickly generate an assortment of finger holders for different sizes of fingers.
CHAPTER 4. ENHANCING STROKE REHABILITATION
58
Figure 4.9: Top view of finger holder
Figure 4.10: The skin protector and wrist brace. Red line represents the embedded metal plate that supports the palm. A positive angle between the palm and forearm, as shown in the figure, can make it easier for the user to grasp objects.
CHAPTER 4. ENHANCING STROKE REHABILITATION
Figure 4.11: Back support and cable guides.
59
CHAPTER 4. ENHANCING STROKE REHABILITATION
Figure 4.12: CAD rendering of cylinder and piston guide assembly
60
CHAPTER 4. ENHANCING STROKE REHABILITATION
61
Figure 4.13: Device for the unimpaired side, showing master cylinder, cable guides, protective sleeve and finger holders.
Figure 4.14: Two different sizes of finger holders for the unimpaired hand.
CHAPTER 4. ENHANCING STROKE REHABILITATION
4.3.3
62
Device Validation
As the prototype was developed, it was taken to patients at Stanford Hospital and the Post-stroke Recovery Program at Foothill College for evaluation. The final prototype was tested by six stroke patients. Although a survey was not conducted, comments from the patients and staff were positive. Patients were impressed with their ability to open their impaired hands using the device. They also reported that it was sufficiently comfortable and adjustable that it could be used for some experiments with haptic feedback, as described in the next section. The main design goals, and the features that contribute to them, are summarized in figure 4.15.
4.4
Adding Haptic Feedback to a Hand-Opening Device
In addition to having difficulty opening the impaired hands, many stroke patients suffer from distorted or greatly diminished haptic sensation in the impaired hand, making it more difficult and less rewarding for them to do exercises. This section introduces a simple haptic feedback system that uses sensors to measure forces at the fingertips of the impaired hand and transmits corresponding feedback signals to the opposite hand.
4.4.1
Pilot Experiment Procedure
Three stroke patients (two male, one female), with ages ranging from 48 to 56, were recruited for pilot experiments. All subjects had the ability to close their impaired hands but could not open them, and all reported no sensation on their impaired hands. While wearing the hand-opening device, the patients were asked to grasp an object, maintain a stable grasp for 10 seconds and replace the object. During this time they were asked to try to minimize the grasp force, without dropping the object.
CHAPTER 4. ENHANCING STROKE REHABILITATION
63
Figure 4.15: Main design objectives and the design features of the portable handopening device that contribute to them.
CHAPTER 4. ENHANCING STROKE REHABILITATION
64
Three different haptic modes were provided: No Haptic Feedback (NHF), Amplitude Based Feedback (ABF) and Event Cue Feedback (ECF). For all haptic feedback modes, the subject’s grasping force was measured by force sensors and haptic feedback was rendered using pager motors. In the ABF mode, the amplitude increased monotonically with the applied force. In the ECF mode, two distinct vibration patterns were used to indicate either an excessive or a dangerously low grasp force. A detailed description of the vibration mapping schemes is provided in Section 5.3, as part of the description of more extensive experiments involving haptic feedback for a group of 24 multiple sclerosis patients. The pilot test was a within-subjects test, so each subject was required to do the task using all three feedback modes. There are six possible orderings of the three feedback modes (NHF-ABF-ECF, NHF-ECF-ABF,etc...). However, for this preliminary test, we only had 3 subjects. Three out of six orderings were chosen and the three different orderings formed an orthogonal matrix. Each subject completed the designed task twice under each of three chosen orderings, following the order shown in table 5.1. For each mode, the grasping task was repeated three times, resulting in a total of 18 trials, with 6 in each mode.
Subject1 Subject2 Subject3
NHF ABF ECF
Three ABF ECF NHF
trials in each ECF NHF NHF ABF ABF ECF
mode ABF ECF NHF
ECF NHF ABF
Table 4.1: Sequences of haptic feedback modes for the 6 subjects Before each experiment, subjects were given time to practice with the three different feedback modes. The pretest practice sessions took from 30 to 60 minutes depending on the individual. Also, whenever subjects switched between feedback modes, they were given several practice trials with the new mode. Subjects could rest at any time during the experiment.
CHAPTER 4. ENHANCING STROKE REHABILITATION
4.4.2
65
External Haptic Feedback
An external haptic feedback channel usually consists of three parts: sensing, signal processing and rendering. In this case, the sensors are force sensors, which measure the grasp force applied by a subject. The sensors are mounted to a plastic rectangular box, as shown in figure 4.16. They measure the normal forces applied to the cover, which wraps around three sides of the box.
Figure 4.16: CAD model of instrumented object consisting of a box with a three-sided cover. The force sensors measure the forces between the cover and the base of the box, from which the normal component of the grasp force can be computed. A signal conditioning circuit filters and amplifies the contact force signal. A filter with a corner frequency at 40 Hz is used to filter out the high frequency force variations and noise. Since the the bandwidth for human grasp control is normally below 10 Hz, especially when users are responding to an external signal, the filter does not remove voluntary variations in force. A PIC micro-controller (PIC18F4431) is used to do the
CHAPTER 4. ENHANCING STROKE REHABILITATION
66
A/D conversion and data interpretation. The controller samples the signal at 200 Hz with a 10 bit A/D resolution. A separate USB DAQ board and a PC computer were also used to collect force data at 1000 Hz for later data analysis. The haptic signal was rendered through two small vibrational motors attached on the back of users’ unimpaired hands.
4.4.3
Experimental Data Analysis
Metrics Two metrics were used to determine the subjects’ performance in controlling their grasping force: the force and the failure rate. In the experiment, subjects were required to maintain a minimum but stable grasping force for 10 seconds. The 10 seconds force data were recorded and used as the first metric. If subjects dropped the object before finishing the 10 seconds grasping task, a failure was recorded. Failure rate is the second metric. The force data from the failed trials were not used in calculating the first metric.
Figure 4.17: Grasping force data with three different haptic feedback modes. To reduce subject to subject variation, data are normalized by the average force without feedback (NHF condition).
CHAPTER 4. ENHANCING STROKE REHABILITATION
67
Results Figure 4.17 shows the force data for three subjects, using each of the feedback conditions. It can be seen that with either ABF or ECF, subjects used less force than with the no-feedback condition (NHF). An ANOVA test was conducted first and statistical significance difference was found (p < 0.0005). Bonferroni corrected paired T tests were conducted to see if there were differences between each pair of modes. Significance was found between NHF mode versus either feedback mode (NHF vs ABF: p < 0.005, NHF vs ECF: p < 0.0002). However no significance was found between ABF vs ECF (p = 0.905).
Figure 4.18: Failure rate with three different haptic feedback modes. Figure 4.18 shows the failure rate for each of the feedback modes. Although we can see some apparent differences in the failure rates, no statistical significance was found in an ANOVA test (p = 0.579).
4.5
Summary and Conclusions
The results from a survey conducted in the Adaptive Learning Division for poststroke recovery at Foothill College provided us with information about the level of hand functionality for people in the program and motivated a system that could make it easier for patients to accomplish basic manual activities earlier in their recovery
CHAPTER 4. ENHANCING STROKE REHABILITATION
68
process. The design objectives for the device were that it should be light, comfortable, portable and easy for patients to put on and take off. Ideally, it should also be passive for simplicity and maximum safety. A hand-opening device to meet these objectives was developed through a series of prototypes and informal evaluations conducted with patients and professionals in stroke recovery therapy. Because stroke patients also typically suffer from reduced or absent haptic sensation, there is an opportunity to further improve the functionality of their hands by providing artificial haptic sensing and remote display. The same approach might also be used on patients suffering from other pathologies, such as multiple sclerosis, that produce a diminished or distorted haptic sensation. The approach introduced in this chapter was to measure the grasping forces as patients lifted and held an instrumented object. The resulting force measurements were used to control two pager motors attached to the back of the opposite (unimpaired) hand. Pilot experiments with three subjects indicated that they could use either proportional feedback or event-cue feedback to control their grasp force. The two feedback modes are analogous to (and inspired by) the proportional and event-cue feedback methods discussed in Chapter 3 for subjects undergoing training in virtual reality environments. Further experiments with additional subjects will be needed to see if the results of the pilot tests are borne out across a sample population of stroke patients. The pilot experiments also suggest that the approach of sensing the forces and providing the feedback can be improved. It would be more versatile to sense the fingertip forces directly, as part of a device attached to the fingers, rather than with a separate sensorized object. Also, given that different fingers may have more or less sensation, it could be effective to provide haptic feedback separately to the corresponding fingers of the opposite hand. An opportunity to test these ideas in more detail was provided through a collaboration with the Masku Neurological Rehabilitation Center in Finland. The patients in this case were multiple sclerosis patients and the details of the experiments are discussed in Chapter 5.
Chapter 5 Haptic Aid for Multiple Sclerosis 5.1
Introduction
The work described in this chapter is an extension of the concept described at the end of Chapter ??, in which signals from force sensors attached to the impaired hand are used to activate vibration tactors (pager motors) on the corresponding fingers on the opposite hand. The work in this chapter took place through a collaboration with Dr. Roope Raisamo at the University of Tampere and Dr. Juhani Ruutiainen at the Masku Neurological Rehabilitation Center in Finland. The experiments were conducted during a two-month period in the summer of 2007, while I was stationed in Finland. The Masku Center focuses mainly on the treatment of patients with multiple sclerosis. Unlike stroke patients, these patients do not require a device to open the impaired hand; therefore only the haptic feedback system was tested. Multiple sclerosis (MS) is an inflammatory disease affecting the human brain and spinal cord characterized by loss of myelin and axons in the nerve tracts. It is the most common cause of neurological disability affecting young adults in the United States and Northern and Central Europe. The symptoms and signs of the disease include motor and sensory dysfunction of the hand and arm [18]. As explained by Dr. Ruutiainen, the dysfunction is often asymmetrically distributed between the left and right upper extremity. The clinical picture differs in severity from one patient to another, yielding individual combinations of reduced modes of sensation, reduced 69
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
70
muscle power and increased muscle tone. Sensory disturbances of the upper extremities are usually related to lesions in the posterior columns of the cervical spinal cord (typically loss of proprioception) but they may also be due to cortical pathology of the brain. Disorder of the autonomic nervous system is relatively common but the peripheral nervous system is usually spared. Realtime feedback has been established as an important consideration for people performing manipulation tasks and learning new motor skills [5, 6, 10]. It has been shown that healthy individuals can accurately control their grasp forces in an exercise such as lifting a glass [27] using information obtained from mechanoreceptors in the fingertips [49, 73, 104, 69]. Diseases such as MS can impair the quality of feedback and the corresponding control of the grasp force [17, 20, 30, 89] so that patients often use larger grasp forces than healthy subjects in tasks that involve grasping and lifting everyday objects [38, 68, 63]. In particular, patients with even mild impairment from MS may show distortions in the coordination of grasp forces when grasping and lifting objects, even when they do not otherwise report difficulty in manipulation [60]. In the case of patients who have reduced or distorted haptic sensation in one limb, there is a tendency to reduce the utilization of that limb and shift tasks to the opposite limb since many tasks are not easy to perform without sensation. The hypothesis behind the work described in this chapter is that patients’ performance in manipulation tasks, and confidence in using the impaired limb, could be increased by providing haptic feedback to the corresponding digits of the opposite limb, leading to increased use of the impaired limb. The technical approach draws upon related investigations in which researchers have tried to improve subjects’ performance in motor related tasks when the normal haptic information channel is either blocked or diminished, by providing sensory augmentation or sensory substitution through vibrotactile and/or force feedback. For example, Murray [71] designed a wearable vibrotactile glove for telemanipulation and evaluated the efficacy of different types of vibrotactile feedback to determine which ones helped subjects to achieve better control of force. Lieberman and Breazeal [62] developed a wearable vibrotactile feedback suit that can detect errors in the motions of a subject’s upper limbs and provide vibrotactile feedback to help them improve
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
71
their performance. In the rehabilitation field, several investigations have addressed providing stroke patients with haptic feedback to the upper or lower limbs [12] [9]. For multiple sclerosis, although many other approaches have been explored [81] [28][1], comparatively little has been done to explore the effectiveness of haptic feedback in response to measured manipulation forces.
5.2
Hardware Design
The portable haptic rehabilitation apparatus is an extension of the feedback system described at the end of Chapter ?? and consists of five parts: force sensors, signal conditioning circuits, micro-controller, amplification circuits and vibrotactile stimulators (Figure 5.1). The computer and DAQ shown in the diagram were used to record force data for later analysis.
Figure 5.1: System Hardware Diagram Initial experiments were conducted with commercial force sensitive resistors (FSRs, InterLink Electronics Inc.). However, difficulties with nonlinearity, drift and hysteresis lead to their being abandoned in favor of a custom fabricated solution. The devices shown in Figure 5.2 utilize low-cost force sensors (FSS1500NST, Honeywell Inc.) embedded in a cast urethane plate. To improve accuracy and decrease the sensitivity of the fingertip force sensor to the point of application of the contact force, the signals from three point-contact sensors are summed to obtain the resultant force on the plate.
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
72
Figure 5.2: Sensor plate with one cent coin. Three sensors are held in the base plate, sensors are not wired in this figure to show a clear layout.
Figure 5.3: Fingers with sensors attached.
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
73
This sensor plate has dimensions of 22mm× 17mm× 4mm and can measure forces up to 44 N before saturating; however in this experiment forces never exceeded 15 N. The measured forces have a linearity within 0.7% and hysteresis of 0.5%. The force signals are amplified and low-pass filtered at 40Hz, which is sufficient given the approximately 10Hz control frequency of human forces in manipulation [92]. A microcontroller (PIC18F4431) monitors the forces with a 10bit A/D resolution for a resulting force accuracy of approximately 0.015N at each fingertip. The controller samples forces at 200Hz and sends corresponding drive signals to the vibrotactile stimulators. To record the data reported in this chapter, a laptop computer with a 12 bit USB data acquisition board (National Instruments USB-6008) also monitored the force signals at 1000 Hz.
Figure 5.4: Experiment setup: force sensors are attached on fingertips of subject’s right hand. Vibrotactile tactors are attached on the fingernails of subject’s left hand. The stimulators are small cylindrical pager motors, 7 mm in diameter by 3mm high. The motors are powered by a 5V DC power supply( 9V battery through a voltage regulator) through darlington transistors which are triggered by the parallel output channels from the microcontroller. The motors have a resonant frequency of approximately 200Hz when taped to a patient’s hand, as shown in Figure 5.4.
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
5.3
74
Experiment Procedure
For patients whose one limb has much worse sensory feedback than the other, there is an opportunity to provide haptic feedback to the less affected limb to improve performance in handling objects. For the experiments conducted in this research, force sensors were attached to the index, middle and ring fingers of a subject’s impaired hand (Instrumenting the thumb was considered, but dropped as it adds little extra information) and vibrotactile stimulators were attached to the back of the fingernail on the corresponding fingers of the unimpaired hand (Figure 5.4). The task in the experiments is inspired by the everyday task of lifting a glass of water. The proxy for the glass is a hollow plastic parallelepiped, 5.7cm × 5.7cm × 15.5cm, weighing 73 grams. We asked subjects to grasp this object and raise it several centimeters from a table top, hold it for several seconds and then replace it. We instructed subjects to apply equal amounts of force on their index, middle and ring fingers when holding the object. When subjects reported that they felt the forces were balanced, we recorded the forces for 5 seconds and then asked them to replace the object. This task was chosen for several reasons. First, the ability to balance the fingertip forces is useful for tasks such as manipulating a glass of water without spilling. The control of grasp and load forces is also a sensitive measure of dexterity and, as reported in [60], deteriorates in mildly impaired patients. Also, because MS patients’ hands are often weak, the use of minimal sums of forces is desirable to decrease fatigue during the task and we found that when patients were able to reduce variations among the forces in their fingers, the overall grasping force tended to reduce as well. As shown in Figure 5.5, our chosen realtime metric also correlates for most subjects (excepting those with the most severe impairment) with the results of the nine-hole pin test. As described further in Section 5.5.1, this is a clinical test used to evaluate dexterity. As a final motivation, the task of balancing finger forces was simple and easy to understand, which allowed all patients to complete the experiment successfully. Subjects conducted the experiments under three different haptic feedback modes:
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
75
Figure 5.5: Corrrelation between the force differences measure and results of the clinical nine-hole pin test. no haptic feedback mode (NHF), amplitude based feedback mode(ABF), and eventcue feedback mode(ECF). The characteristics of each feedback mode are discussed in the next section. Three haptic feedback modes were tested: no haptic feedback (NHF), event-cue feedback (ECF) and amplitude based feedback (ABF). The characteristics of each mode are discussed in the next sections. 24 multiple sclerosis patients were recruited as subjects at the Masku Neurological Rehabilitation Center in Finland. Eight of those subjects are males, sixteen are females. The range of ages is from 33 to 64 with a mean of 56.4. The recruited subjects all have reduced sensation in one hand and good sensation in the other hand. They were all able to fully understand the human consent form and to follow the simple instructions required to complete the sessions.
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
76
Due to the large number of subjects required for a between-subjects test, a withinsubjects test was chosen. To mitigate learning effects associated with a withinsubjects test, two measures were taken: long practice sessions and randomized test orders. Before the tests, subjects were given time to become familiar with the different haptic feedback modes until no further improvement in the force metric was observed. The pretest practice sessions took from 30 to 60 minutes depending on the individual. Then a practice session, which was identical in length and format to the experiment session, was used to help subjects get familiar with the task and reduce learning effects. Immediately after the practice session, the experiment session started. With three feedback modes, there are 6 possible orderings in which the modes can be presented to the test subject. (NHF-ABF-ECF, NHF-ECF-ABF, etc... ). During the experiments, all 24 subjects completed the task 3 times for each feedback mode, in a sequence of 9 feedback modes, resulting in a total of 27 trials (see table 5.1). The sequence of feedback modes was chosen randomly from the set of 6 possible orderings. At the start of each new mode, subjects were given several practice trials before collecting data.
Subject1 Subject2 Subject3 Subject4 Subject5 Subject6
NHF NHF ABF ABF ECF ECF
ABF ECF NHF ECF ABF NHF
Three trials in each mode ECF NHF ABF ECF NHF ABF NHF ECF ABF NHF ECF ABF NHF ECF ABF NHF ABF ECF NHF ABF NHF ECF ABF NHF ECF ABF ECF NHF ABF ECF
ABF ECF NHF ECF ABF NHF
ECF ABF ECF NHF NHF ABF
Table 5.1: Sequences of haptic feedback modes for the first 6 subject
5.4
Force to Vibration Mapping
Our device measures contact force information on a subject’s impaired hand and renders the information back the unimpaired hand using vibrations. As mentioned in
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
77
the previous section, three different haptic feedback modes were used in the experiment : no haptic feedback mode (NHF), amplitude based feedback mode(ABF), and event-cue feedback mode(ECF). This section discusses the force to vibration mapping in ECF and ABF modes.
5.4.1
Event cue feedback
As seen in Chapter 3, event cue feedback is a simple technique in which transient vibratory stimuli are imparted only when a certain condition occurs. In the present case, vibratory stimuli are applied only when the fingertip forces become unbalanced, i.e. when differences between the magnitudes of the forces exceed a threshold: if (max(fi ) − min(fi ) > fr ) then apply vibration where i = 1, 2, 3 for the index, middle and ring fingers, respectively and fr is a threshold that is adjusted empirically during the practice trials for each subject. Generally more impaired subjects need higher fr , while slightly impaired subjects can have lower fr . On average, fr is approximately 0.7N. Whenever vibration is to be displayed, the next question is where to direct it. The vibration is applied to whichever finger deviates most from the average force, fa = (f1 + f2 + f3 )/3, and is of Type I (high force) or Type II (low force) depending on whether the associated finger is above or below the average. The Type I stimulus consists of regular pulses at 35 Hz with a 50% duty cycle; the Type II stimulus consists of pulses at 1.33 Hz with a duty cycle of 10%. These parameters were determined empirically in pilot tests and found to be noticeable and easily distinguishable. The result of this approach is that when the fingertip forces are approximately equal, there is no stimulus. If one of the fingers drops substantially below the average value, a Type II (low force) vibratory cue is applied to that finger until it returns to within fr of the mean. Conversely, if one of the fingers applies an excessive force, a Type I (high force) cue is applied until it returns to within fr of the mean.
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
5.4.2
78
Amplitude based feedback
The amplitude based feedback (ABF) is based on mapping the intensity of vibration at each fingernail to the magnitude of the force at the corresponding opposite finger. After several pilot tests involving variations of continuous amplitude and frequency, and pulses of varying frequency and duty cycle (i.e., the fraction of each period that a motor is turned on), the following scheme was adopted: for each finger, if the measured force exceeds a threshold of 0.05N, the microprocessor commands a vibratory stimulus. The vibration takes the form of a train of pulses applied to the pager motor such that the pulse frequency and duty cycle both increase with increasing force. The period, T in seconds is specified by C1 (5.1) Fi + C2 where C1 = 0.3, C2 = 0.25 and Fi is the force, in N, on i’th finger. The pulse width, T =
Pw , in ms is then given by ( Pw =
0
if Fi < 0.05N
10 + 5n if Fi ≥ 0.05N
(5.2)
where n = min(f loor(Fi /0.75N ), 6). Thus at 0.05N, the frequency of pulses is 1Hz and the duty cycle fraction is (0.01/1), corresponding to 10ms during each period when the vibrator is on. For larger forces, the period decreases and the duty cycle fraction increases as shown in Figure 5.6, ultimately saturating at a force of 7N. A few patients occasionally produced forces as high as 15N; however to improve the resolution for normal handling forces of 1-4N, the upper bound was taken as 7N. Any larger forces are the result of large transient errors and little information is lost by saturating them at 7N. (Note that for light forces, the pulses are short enough that the motor never reaches its steady-state speed.)
5.4.3
Perception Test of Amplitude Based Feedback(ABF) Mode
For the amplitude based feedback, it is important to verify that subjects are able to map the vibration stimulus to force levels with sufficient accuracy and repeatability
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
1
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
0
2
4 Force (N)
6
Period (s)
Duty Cycle
1
79
0 8
Figure 5.6: Period length and duty cycle versus force, in the amplitude based mapping method (ABF). to improve their grasping performance. In previous studies, it was found that better dynamic range could be achieved by varying amplitude and frequency together rather than varying either amplitude or frequency alone [71]. In the present case, because vibration is provided by pager motors, amplitude and frequency cannot be controlled independently. The approach of varying the period and duty cycle of pulses is an effort to overcome this limitation. Tests were first conducted to establish the correlation between different values of the stimulus pattern and the perceived magnitude of a corresponding quantity (which would ultimately be the measured grasp force in the final application) and to determine the resolution with which they could distinguish between different levels of vibration intensity. Perceived Intensity Test Six subjects were recruited for the first perception test, which was a form of magnitude estimation with free response [108]. The subjects had no sensory impairment on the tested hand. In the experiment, nine different levels of vibration stimulus were presented, using the settings given in Figure 5.6. In the experiment, subjects were first presented with the maximum and minimum levels of the vibration stimulus, which could be repeated as many times as they desired. After each stimulus, subjects were
80
Perceived Strength
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
Vibration Level’s Corresponding Force (N) Figure 5.7: Average perceived magnitude, normalized for each subject, corresponding to the periods and duty cycles in equation 5.1 and 5.2 associated with different force levels. Error bars show standard error.
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
81
asked to assign a number. No scale was specified, but zero was excluded. Subsequent stimulus levels were presented in random order and each stimulus level was presented ten times for a total of 90 stimuli per subject. However, the first 45 trials were used as practice and only data from the last 45 trials were recorded. To analyze the data, the subjects’ responses were normalized following the procedure in [108]. Each subjects’ response was normalized by dividing it by the subject’s mean for a particular vibrational level and then multiplying by the total mean for the given condition for all subjects: Figure 5.7 shows average perceived magnitude for each stimulus level versus the corresponding nominal force magnitude. Error bars show the standard error. The trend is monotonic and approximately linear, which suggests that subjects can map the variations in stimulus to corresponding variations in a quantity such as force. It can also be seen that when vibration is light, the standard deviation of the perceived magnitude is smaller. Interestingly, there is a change in slope at the third stimulus level, corresponding to a nominal force of approximately 1.5N. This change corresponds to the condition under which the motor starts to achieve its full rotation speed during pulses; below this level, the pulses are too brief for the motor to spin up fully. Resolution Test A second experiment was conducted to determine the minimum resolution for distinguishing between different vibrational patterns presented simultaneously on two fingernails. For this test, five different levels of stimulus, corresponding to nominal force magnitudes between 0.75 N and 3.75 N were tested, as these cover the range of force used by most subjects in handling the object. In the test, a reference vibrational stimulus corresponding to one of the five forces was presented to the subjects’ middle fingernail. The vibration was presented for 1 second followed by a 1 second pause. Then subjects were asked to adjust the vibration stimulus on their index fingernail (by using two keys on the keyboard) to match the magnitude of the vibration on their middle fingernail. When they reported the stimuli were matched, the difference was recorded. The order of the reference signals was randomized and each test point was presented to subjects three times for a total of 15 trials. The same six subjects
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
82
participated in this test as in the perceived intensity test and their results are plotted in figure 5.8.
Figure 5.8: The average minimum detectable difference between vibration stimuli corresponding to different nominal force values between 0.75 N and 3.75 N. The detection curve has a convex shape in Fig. 5.8. It reaches a minimum for a vibration stimulus level that corresponds to a nominal force of 2.25 N. Again, this maximum sensitivity appears to occur when the motor has started to achieve its full speed during pulses. Also, as in the perceived magnitude test, the variability among subjects is smaller for low stimulus levels. Discussion From the results of the two experiments, we can see that by using the pulse-width and pulse-interval modulated approach, the mapping from grasping force on the impaired hand to perceived magnitude of vibration is monotonic and roughly linear. In Figure 5.7, all data points were connected with lines. There is a change in slope at approximately 2N, corresponding to the vibration level for which the motor reaches full rotation speed during pulses. It was also found that the greatest sensitivity to changes in force occurred at this condition. A force of 2N also corresponds to typical handling forces in the experiments.
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
5.5
83
Experiment results
Two metrics were used to determine the subjects’ performance in the force balancing experiment. The primary metric chosen for the experiments was the degree to which subjects could achieve an even distribution of grasping forces among the fingers. This was measured by the sum of forces (forces on index, middle and ring fingers) difference to the average force. The secondary metric is the failure rate, which is determined by how many times that subjects unintentionally dropped the object from their hand. Although subjects were not asked to conduct the task quickly or slowly, task completion time was recorded to see if there is any correlation between completion time and task performance. After finishing experiments, all subjects were asked to complete a survey with 4 questions and data were collected to learn their subjective impressions of the haptic feedback modes provided in the experiments.
5.5.1
Force Control Metric
Force Control for All Subjects The force balancing metric is computed as fs =
3 X
abs(fi − fa )
(5.3)
i=1
where fa is the average force and, again, i = 1, 2, 3 for the index, middle and ring fingers, respectively. Subjects were asked to maintain an even force balance for 5 seconds after lifting the object and the value of fs was recorded continuously during this time. If the subject dropped the object or was unable to hold the object for 5 seconds, a failure event was noted. Figure 5.9 shows typical plots of forces on index, middle and ring finger, with amplitude based feedback and without haptic feedback. To reduce the effects of subject to subject variability, all force data from each subject are normalized by the subject’s average value without haptic feedback. An ANOVA test was conducted and showed significant differences among the three feedback modes ( F (2, 645) = 298.13, p = 0). Further, Bonferroni multiple comparisons
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
84
Figure 5.9: Typical force plots with ABF (amplitude based feedback) mode and with NHF (no haptic feedback) mode were conducted. As seen in Figure 5.10, there is a significant difference in the force variations, fs , when using ABF or ECF (p < 1 · 10−10 , p < 1 · 10−10 respectively compared with NHF, using a Bonferroni corrected paired T test [24]). However no significant difference was found between the ABF and ECF modes (p < 0.27 in Bonferroni corrected T test). An ANOVA test of the standard deviation also showed significant differences among the three modes (F (2, 69) = 42.13, p < 1.1 · 10−12 ). Significant reduction (σ = 0.17 for ABF and σ = 0.16 for ECF) as compared to the no-feedback case (σ = 0.38, ) was found in further Bonferroni corrected paired T tests (p < 3.9 · 10−7 , p < 2.1 · 10−7 respectively). Correlating the effect of feedback with the degree of impairment Although the overall data for 24 subjects do not show a significant difference between the ABF and ECF modes, we noticed during the experiments that subjects with greater impairment seemed to prefer the proportional feedback mode (ABF) while those with less impairment preferred event-cue feedback (ECF). Accordingly, we divided the subjects’ data into two groups based on their level of impairment, to look for a correlation between impairment and the improvement achieved using either
Sum of Force Difference to the Average Force
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
85
Sum of Force Difference in 3 Different Modes 2.5
2
1.5
outliers std deviation
1
75%
0.5
mean 25%
0
NHF
ABF
ECF
Figure 5.10: Sum of force differences, fs , under NHF(no feedback), ABF(amplitude) and ECF(event cue) feedback modes. Both ABF and ECF result in significantly smaller variations in force compared to NHF; no significant difference is found between ABF and ECF. (Data for each subject are normalized by the subject’s average value under NHF.) ABF or ECF. The subjects had all previously been evaluated using a standard clinical test. The 9-Hole Peg Test [32] is a quantitative measure of upper extremity function, widely used in MS clinical trials. It is one of the components of the Multiple Sclerosis Functional Composite that measures three important clinical dimensions of the disease, namely arm function, leg/walking function and cognition [85]. The test consists of moving nine pegs into one of nine holes on a peg board, then back into an open box. Subjects are scored in terms of the time required to complete the test, as compared to the normal range of times for subjects in the same age group. Published data indicate that results for unimpaired subjects are approximately normally distributed around the mean value for each age range.
86
ABF better
30 20 10 0 ECF better
Difference in percent improvement for ABS vs ECF
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
-10 -20 -30 -40 -1 10
0
10 Impairment level, IL (eq. 3)
10
1
Figure 5.11: Difference in percent improvement for ABS vs ECF with respect to NHF mode. Each point corresponds to one subject. The x coordinate in log scale shows subjects’ impairment level, IL . A line shows the best log fit to the data. For each subject, we computed an impairment level, IL as follows: IL =
TS −1 TN
(5.4)
where TS is the time taken by the subject to complete the 9-Hole Peg Test and TN is the average time for unimpaired individuals in the subject’s age group. Thus, a value of IL = 0 indicates no impairment and larger values indicate increasing impairment. For 23 of the 24 subjects, the value of IL ranges from 0.25 to 4.53. (One subject was not able to finish the 9-Hole Peg Test test, so his value of IL would be infinite.) We then computed the difference in improvement obtained with ABF versus ECF over the baseline no-feedback (NHF) case. Figure 5.11 shows the results for the 23 subjects. The horizontal axis measures the impairment, IL , plotted on a log scale. The vertical axis measures the difference in the percentage of improvement
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
87
(in fs ) for the ABF versus the ECF feedback case. Subjects who showed the most improvement over the no-feedback case when using amplitude based feedback (ABF) are in the upper half of the plot and subjects who showed greater improvement with event-cue feedback (ECF) are in the lower half. A glance at the figure shows that the majority of less-impaired subjects (IL < 100 ) are in the lower half of the plot while the more impaired subjects are mainly in the upper half. A best fit line is also plotted to the data. While the data show considerable scatter with respect to the line, the basic trend of greater improvement obtained with ABF vs ECF, with increasing impairment, is evident. Moreover, the relative improvement with ABF vs
Sum of Force Difference for Less Impaired 2.5
2
1.5
1
0.5
0
NHF
ABF
ECF
Sum of Force Difference to the Average Force
Sum of Force Difference to the Average Force
ECF appears to increase approximately the impairment level. Sum of Force logarithmically Difference Lesswith Impaired Sum of Force Difference for More Impaired 2 1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0
NHF
ABF
ECF
Figure 5.12: Force variation, fs , for different feedback modes, with subjects divided into groups based on the degree of impairment. In both groups, the ABD and ECF feedback modes show improvement over the no-feedback (NHF) case. In the lessimpaired group, the ECF mode was significantly better than ABF (p < 1 · 10−5 ); in the more impaired group the reverse was true(p < 1 · 10−5 ). The results in Figure 5.11 suggest that if we divide the subjects’ data into two pools of less-impaired and more-impaired subjects, we should find significantly different
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
88
levels of improvement with ABF versus ECF. Figure 5.12 shows the results of this division. We divided the subjects’ data into two groups, depending on whether their IL value was greater or less than 1. This division is somewhat arbitrary, as the subjects’ IL values are relatively evenly distributed on a logarithmic scale, as seen from Figure 5.11. However, the results are not much affected by whether the cutoff is at IL = 0.8 or IL = 1.3 instead of 1. The results of dividing the subjects into two groups are shown in Figure 5.12. As in Figure 5.10, the measure of performance is the force imbalance, fs from eq. 5.3. The less-impaired group contains 10 subjects and the more-impaired group contains the remaining 14. First, ANOVA test was performed in both less and more impaired groups and significant differences were found (Less impaired: F (2, 267) = 174.37, p = 0, More impaired: F (2, 375) = 153.14, p = 0). For the less-impaired group, the ECF mode provides significantly more improvement (p < 3 · 10−5 using a Bonferroni corrected paired T test) than the ABF mode. For the more-impaired group, ABF provides significantly better performance than ECF (p < 4 · 10−5 ).
5.5.2
Comparison with unimpaired subjects in a force balance task
The impairment level for the multiple sclerosis subjects recruited in the experiment varied over a wide range, and correlations between the feedback modes and task performance were found in the previous data analysis. For comparison, six unimpaired subjects were also tested to see whether the haptic feedback modes would produce a difference in the ability to control grasp forces. The six healthy subjects included two females and four males, with ages ranging from 24 to 30. All were right handed. Three conducted the experiment with their left hands and three used their right hands. Initially, the subjects were asked to do the same task as the multiple sclerosis subjects. The sums of the force differences were calculated using the same method as in the previous experiment for MS subjects. The results are shown in Figure 5.13. Following the procedure used in Fig. 5.10, data for each subject are normalized by the subject’s average value under no-feedback
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
89
Sum of Force Difference to the Average Force
Sum of Force Difference for noneimpaired in none moving case 2.5
2
1.5
1
0.5
NHF
ABF
ECF
Figure 5.13: Sum of force differences of unimpaired subjects for the same task as MS subjects under the different feedback modes (NHF: no haptic feedback, ABF: amplitude based feedback, ECF: event cue feedback). Using a Bonferroni corrected T test, no significance was found between NHF and ABF or between ABF and ECF. However significance was found between NHF and ECF. (Data for each subject are normalized by the subject’s average value under no haptic feedback.)
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
90
to reduce subject-to-subject variability. As one might expect, haptic feedback is much less useful for these healthy subjects, and the differences among the NHF, ABF and ECF modes are not significant at 95% confidence level (ANOVA test: F (2, 159) = 2.85, p < 0.0609). A further experiment was conducted with the same six subjects with an increased level of difficulty to see if the effects of haptic feedback could be observed. In this experiment subjects were asked to raise the object from the desk first and then balance the forces among their index, middle and ring fingers. After they felt the force was balanced, they needed to move the object over a cardboard barrier 30 cm high on the desk and place the object on the other side of the cardboard. While moving,
Sum of force difference while five seconds were recorded. moving object
subjects were still required to balance the forces on the three fingers. The time taken
Sum of Force Difference to the Average Force
to move the object took approximately 5 seconds, and the force data during the last
3
2.5
2
1.5
1
0.5
0
NHF
ABF
ECF
Figure 5.14: Sum of force differences for unimpaired subjects balancing forces while moving an object over a 30 cm high obstacle. Significance was found between each pair of feedback modes. (NHF: no haptic feedback, ABF: amplitude based feedback, ECF: event cue feedback). (Data for each subject are normalized by the subject’s average value under no haptic feedback.)
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
91
Figure 5.14 shows the sum of force differences in the second experiment for healthy subjects. In this task, force variations are the smallest in ECF mode, followed by the ABF mode, consistent with the previous results for mildly-impaired MS patients in a simpler task (Fig. 5.12). An ANOVA test shows significant differences among the three different modes (F (2, 159) = 50.13, p ≈ 0). Bonferroni corrected paired T tests
Failure Rate
were conducted and significant differences were found between each two mode pairs (HNF vs ABF: p < 2 · 10−6 HNF vs ECF: p < 1 · 10−8 ABF vs ECF: p < 1 · 10−7 ).
5.5.3
Failure Rates for MS patients 7
Failure Rate
6 5 4 3 2 1 0
NHF
ABF
ECF
Figure 5.15: The numbers of failures per subject for 24 MS patients under the three different feedback modes are statistically different. ABF provides the lowest failure rate. For multiple sclerosis patients, the failure rate (i.e., the number of times that an object was dropped or could not be held for 5 seconds in each set of three trials with a given feedback mode) is another measure of performance. As seen in Figure 5.15, the baseline NHF mode has the highest failure rate and the amplitude-based ABF mode has the lowest. A one-way ANOVA test was conducted and showed significantly different results (p < 1.5 · 10−5 ) among the modes. Bonferroni corrected T tests indicate that each pair of modes is statistically different: NHF vs ECF, p < 8.08·10−5 ;
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
92
NHF vs ABF p < 1.5 · 10−9 ; ABF vs ECF, p < 0.003.
5.5.4
Task Completion Time for MS patients
Although subjects were told there was no time requirement, the task completion time was recorded to see if there was any correlation between haptic feedback mode and task completion time. The task completion time was measured as the time from when the object lifted from the table until it was replaced. Figure 5.16 shows the subjects’ task completion time under three different modes. To reduce subject to subject variation, time was normalized by each subjects’ average completion time under NHF mode. It can be seen that subjects used least time under NHF mode and used the longest time in ECF mode. A one-way ANOVA test was conducted and significant difference was found among the results (p < 4.9·10−5 ). Paired T tests with Bonferroni correction show significance between each pair of modes: NHF vs ECF, p < 1 · 10−10 ; NHF vs ABF p < 7 · 10−8 ; ABF vs ECF, p < 0.014. In other words, subjects became both slower and more accurate when they were paying attention to the haptic feedback. To see if there was a correlation between impairment level and task completion time, average task completion times (with respect to NHF mode) for each subject in ABF mode and ECF mode were plotted against impairment level in Figure 5.17. Each dot or small circle in the plots represents one subject. The X coordinate of the dot shows the impairment level and the Y coordinate shows the completion time, normalized with respect to the NHF condition. Best fit lines are also shown. Evidently there is not a strong correlation between completion time and impairment level. To check further if there was a correlation among impairment level, feedback mode and task completion time, the differences in task completion time for ABF versus ECF (with respect to NHF) were plotted against the impairment level in Figure 5.19. Again, each circle represents one subject. Circles above the zero line represent subjects who used less time with ECF mode, while circles below zero represent subjects who used less time with ABF. It can be seen that when the impairment level, IL , is smaller than 1.3 the circles are mostly centered around zero, indicating
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
93
Figure 5.16: Task completion time under three different modes. Time was normalized by each subject’s average completion time in NHF mode.
Figure 5.17: Task completion time in ABF mode and ECF mode versus impairment level. Each dot or circle represents one subject. Best fit lines are plotted to the data.
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
94
¡¡¡¡¡¡¡ .mine
Figure 5.18: Difference in completion time for ABS vs ECF with respect to NHF mode. Each circle represents one subject. The X coordinate shows subjects’ impairment level, IL . =======
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
95
little difference. However, for the more impaired subjects, the ABF mode consistently produced a greater reduction in task completion time. Finally, it appears that the relative improvement was perhaps greatest for patients with moderate impairment (IL ≈ 1.5).
5.5.5
Experiment survey for MS patients
The data from the experiments suggest that less-impaired patients perform better with the event-cue (ECF) feedback whereas more-impaired feedback perform better with amplitude-based (ABF) feedback. A post-test survey of all the MS patients was conducted to assess their subjective impressions of the different modes. The survey was conducted in Finnish, with the assistance of a therapist at the center. Four questions were asked of each subject and a translation of the results follows. Question one: Under which mode do you believe you performed best? (NHF, ABF, ECF or not sure) The first question was designed to determine whether subjects’ impressions correlated with their actual performance. We asked each subject to vote for the mode under which they performed best. We compare this with their actual best mode in Table 5.2. Subjects are grouped into less-impaired and more-impaired subgroups, as in Figure 5.12. More impaired (14) Votes Correct Votes on NHF mode 3 0 Votes on ABF mode 7 7 Votes on ECF mode 4 0 Correct Rate 50%
Less impaired (10) Votes Correct 2 0 0 0 8 7 70%
Table 5.2: Question One results: Votes indicate how many subjects believed they did best with the corresponding mode; Correct votes are those that match the actual best mode for the same subjects.
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
96
From Table 5.2, we can see that, as might be anticipated, less-impaired subjects were slightly more accurate in identifying the feedback mode under which they performed best. Question two: Which feedback did you prefer, ABF mode or ECF mode? In the second question, we asked subjects about their preferred mode, independent of whether they thought they performed best with it. Results are summarized in Table 5.3. More impaired ABF mode preferred 8 ECF mode preferred 6
Less impaired 1 9
Table 5.3: Preferences for ABF versus ECF modes From Table 5.3 we see that less impaired patients had a strong preference for the ECF mode. Question three: Rate the helpfulness of ABF mode and ECF mode. Question three was designed to assess the perceived helpfulness of the ABF and ECF modes. Subjects were asked to rate the helpfulness of each mode separately on an integer scale from -3 to +3, with -3 indicating that the mode significantly hindered performance, 0 indicating no effect and +3 indicating significant helpfulness. Average ratings for each group are shown in Table 5.4.
ABF avg. score ECF avg. score
More impaired 2.07 2.07
Less impaired 1.80 2.50
Table 5.4: Average helpfulness ratings for ABF and ECF modes (-3 = substantial hindrance, 0 = neutral, +3 = substantial help).
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
97
Both modes were viewed as helpful by all subjects (all ratings > 0) with the highest average rating of 2.5 for less-impaired patients using the ECF mode. The more impaired subjects rated the ABF and ECF modes as equally helpful (although they performed better with ABF). Question four: Rate the ease of learning for using the ABF mode and ECF mode. Subjects rated the ease of learning using the same -3 to +3 integer scale as in the previous question, with -3 indicating quite difcult and +3 indicating quite easy. Average ratings for each subject group are shown in Table 5.5.
ABF avg. score ECF avg. score
More impaired 1.86 1.36
Less impaired 0.80 1.70
Table 5.5: Average perceived ease of learning for ABF and ECF modes on a scale from -3 (quite difficult) to +3 (quite easy). As seen in Table 5.5, the more impaired patients found the ABF mode slightly easier to learn whereas the less impaired patients found the ECF mode substantially easier to learn. The reasons for this discrepancy have interesting implications for future implementation of haptic feedback systems for MS patients and are dis- cussed in the next section.
5.6
Discussion and Conclusion
A simple system that includes force sensors at the fingertips of the impaired hand and vibration pulses applied to the fingernails of the opposite hand can improve the performance of MS patients in a grasping and lifting task. The results of the preceding sections indicate that either amplitude-based feedback (ABF) or event-cue feedback (ECF) can help subjects to balance their grasping forces in handling objects.
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
98
Normalizing each subject’s results by their no-feedback performance reduces the subject to subject variability sufficiently to reveal further significant trends. In particular, we found that subjects with mild impairment performed best with event-cue feedback while subjects with severe impairment performed better with proportional feedback. This finding also generally matches the preferences voiced in the survey of the subjects. A couple of reasons may explain this effect. First, the proportional feedback is always on and can become annoying when it is rarely needed by patients with mild impairment. In contrast, the event-cue feedback is only triggered when a subject is in danger of dropping or bobbling the object, perhaps due to fatigue or a momentary distraction. However, for subjects with severe impairment, the event-cue feedback is triggered frequently and can become distracting. These patients have relatively little sense of how much force their fingers are providing, and proportional haptic feedback provides a straightforward and continuous measure of their activity. For comparison, similar tests were conducted with healthy subjects. Initially, the healthy subjects showed no significant effect due to haptic feedback. This is perhaps to be expected, and is consistent with the concept that variations in grasp forces are an useful indicator of impairment. After increasing the difficulty of the manipulation task, healthy subjects showed slightly improved performance with haptic feedback, particularly for the event-cue mode. This result is consistent with the results of the less-impaired MS patients. We also observed also that the the event-cue feedback produced more failures than the proportional feedback (but fewer failures than no feedback). This result may be a reflection of the delay involved in detecting a dangerous situation (unbalanced forces), alerting the user and allowing the user to respond in time to prevent dropping the object. Providing an earlier warning of impending failures may reduce the failure rate with ECF. These findings are generally consistent with observations in other applications where a natural haptic feedback channel may be absent. For example, in experiments involving dexterous teleoperation of a slave robot [35] it was found that proportional feedback to the operators generally gave better performance than event-cues (e.g. grasp force too low or too high). The conclusion to draw may be that when a natural
CHAPTER 5. HAPTIC AID FOR MULTIPLE SCLEROSIS
99
feedback channel is absent, it is best to replace it with a proportional artificial one, but when an existing channel is active (perhaps with diminished effectiveness) it is better not to add a duplicate channel and instead to alert the user with cues of impending events. The next step in this work is to determine whether lasting rehabilitation takes place as a result of utilizing the feedback. This will require an extended series of tests over a period of time. Our hope is to utilize the plasticity of the brain to compensate for central nervous system damage that is a primary mechanism of functional recovery in MS [82]. Since a similar phenomenon occurs in other important diseases of the central nervous system such as brain injury and stroke, the results may be of wide importance in neurological rehabilitation of hand and arm function. If rehabilitation can be shown, it would not be difficult to develop a miniaturized and more robust version of the apparatus that patients could use daily in their homes.
Chapter 6 Conclusions The focus of the work presented in this dissertation is the development and evaluation of portable haptic devices that utilize vibration feedback, to provide either proproptional feedback whose amplitude is directly related to a quantity such as force, or an “event-cue” i.e., a notification that some event such as contact or a significant change in forces has occurred. The applications include training of procedures in a virtual reality environment and neurological rehabilitation. In both cases, an issue is that normal haptic feedback channels are distorted or absent. Providing artificial haptic cues allows people in both cases to perform tasks with fewer errors. Also, in both cases, training is evident: people get better at utilizing the artificial haptic feedback with practice. The following sections summarize the findings from this work and discuss suggestions for future work. Virtual reality environments are increasingly used for training procedures for teams of emergency and military personnel. An extension to the visual and audio environment that a standard VR system can provide is to incorporate haptic feedback, so that the user can feel virtual contacts, changes in loads, etc. However, the requirements of providing realistic force feedback to a mobile individual are daunting and require large, complex and expensive systems. An alternative is to focus on providing cues about the timing and approximate magnitudes of events such as contact forces. In the experiments described in Chapter 3, two kinds of approximate haptic feedback were explored. The first involved vibrations associated with contact 100
CHAPTER 6. CONCLUSIONS
101
events. The second involved simplified forces, approximately equal to the impulse that would be expected from a contact with a hard surface while moving at a certain velocity. In both cases, the addition of these haptic events allowed users to make fewer procedural errors in a simple training scenario. Thus, as with visual and audio feedback, it appears that haptic feedback in a virtual reality environment does not need to be highly realistic for benefits to accrue. A natural extension of the work in Chapter 3 would be to explore wearable, wireless tactors for multiple people in a training exercise, perhaps adapting the technology used in [62]. Inspired in part by the success of the simple experiments in Chapter 3, I was interested in exploring whether similar portable, and perhaps wearable, feedback devices could be used to help individuals suffering from pathologies that affect haptic sensation. In the case of the virtual reality environment, the induced haptic feedback roughly simulates the feedback that individuals would normally receive. In the case of stroke and multiple sclerosis patients, an artificial haptic feedback channel augments a natural one that has become significantly distorted or diminished. As in the case of virtual reality training, most previous rehabilitation devices have been comparatively bulky, complex and expensive. A low-cost, robust and wearable system could potentially be supplied to users to take home with them. Stroke and multiple sclerosis patients often have one side affected and the other side relatively intact. Through a series of prototypes and evaluations, I arrived at a purely passive mechanical master/slave system in which stroke patients could use their unimpaired hands to open their impaired hands, for example, to grasp an object. To this device I added a haptic feedback system that measured the grasp forces exerted by the impaired hand and transitted vibration cues to the back of the opposite, unimpaired hand. Pilot tests with three stroke patients indicated that this system could help them to regulate their grasp forces. Additional testing is needed to confirm the efficacy of the system and to determine whether it aids in long-term recovery from stroke. However, in the interim, I was given the opportunity to conduct an extensive set of experiments on a group of 24 multiple sclerosis patients in Finland. A modified version of the haptic feedback system was applied to the multiple sclerosis patients. Force sensors measured grasp forces at the fingertips of the impaired
CHAPTER 6. CONCLUSIONS
102
hand. Small vibration tactors applied stimuli to the fingernails of the fingers on the opposite hand. With this system, the multiple sclerosis patients were able to regulate their grasp forces more accurately and with fewer errors (e.g. dropped objects). In additional tests I established that patients were able to map between levels of vibration stimulus and corresponding levels of force and that they were able to distinguish between two types of events corresponding to excessive and dangerously low levels of force, respectively. Both the proportional feedback and the event-cue feedback modalities were effective. However, I found that patients with slight impairment preferred, and did better, with the event-cue feedback whereas the severely impaired patients performed better with proportional feedback. The conclusion to draw from this phenomenon may be that when a natural feedback channel is absent, it is best to replace it with a proportional artificial one, but when an existing channel is active (perhaps with diminished effectiveness) it is better not to add a duplicate channel and instead to alert the user with occasional cues of impending events.
6.1
Summary of Contributions
The work described in this dissertation has provided several contributions to the field of virtual reality training, the field of stroke rehabilitation and field of multiple sclerosis rehabilitation. The main contributions covered by the dissertation are as follows: • In this thesis I have shown that simple, portable haptic feedback devices can improve the performance of individuals undergoing training in a virtual reality environment. The feedback does not need to be particularly realistic; it can consist primarily of feedback about the occurrence of events such as contacts. • I developed a novel portable hand open rehabilitation device for stroke patients that does not require external power. The device allows stroke patients to open their impaired hands using power from the opposite hand and allows them to grasp objects with the impaired hand. • I augmented the stroke rehabilitation device with a simple haptic feedback
CHAPTER 6. CONCLUSIONS
103
system that measures grasping forces applied by the impaired hand and transmits corresponding vibration signals to the opposite hand. The system shows promise for future work on stroke rehabilitation. In pilot experiments, stroke patients were able to control their grasp force better with the haptic feedback than without. • I demonstrated that a haptic feedback system based on measured grasp forces is also useful for multiple sclerosis patients. The results of experiments on 24 multiple sclerosis patients showed that with extra haptic feedback, they achieved significantly better grasp force control during manipulation tasks. • I discovered that for multiple sclerosis patients, there is a correlation between the level of impairment and the type of feedback that is most effective. Severely impaired patients did better with proportional feedback; mildly impaired patients did better with occasional event cues.
6.2
Future Work
This thesis has shown that portable and wearable haptic feedback systems can improve subjects’ performance in a variety of applications. The details of how to display the feedback depend on the application and on the condition of the individual to which feedback is provided. Following along the lines of the work presented in this thesis, several extensions are possible. In particular, although low-cost, simplified haptic feedback was found to be effective, the relative efficacy of such feedback was never established in comparison with the results obtained with a high-end system that provides a more accurate force and/or vibration display. A quantitative comparison would help to determine the cost/benefit tradeoff in providing haptic feedback. In addition, while the systems described in this thesis provided some immediate improvement in hand functionality for stroke and multiple sclerosis patients, the long term effects are unknown. Will such systems encourage patients to use their impaired hands more and, if so, will they recover functionality faster?
Bibliography [1] S. A, G. Filippini, P. Gasco, L. Colla, A. Salmaggi, L. La Mantia, and et al. Physical rehabilitation has a positive physical rehabilitation has a positive effect on disability in multiple sclerosis patients. Neurology, 52:57–62, 1999. [2] H. Ando, T. Miki, M. Inami, and T. Maeda. The nail-mounted tactile display for the behavior modeling. In SIGGRAPH ’02: ACM SIGGRAPH 2002 conference abstracts and applications, pages 264–264, New York, NY, USA, 2002. ACM. [3] K. Bark, J. Wheeler, S. Premakumar, and M. Cutkosky. Comparison of skin stretch and vibrotactile stimulation for feedback of proprioceptive information. In Haptic interfaces for virtual environment and teleoperator systems, pages 71–78. Haptics 2008, March 2008. [4] J. T. Bell and S. H. Fogler. Low cost virtual reality and its application to chemical engineering - part one. Computing and Systems Technology Division Communications, American Institute of Chemical Engineers, 18(1), 1995. [5] E. Bilodeau and I. Bilodeau. Some effects of introducing and withdrawing knowledge of results early and late in practice. J. Exp. Psychol, 58:142–144, 1959. [6] E. Bilodeau and I. Bilodeau. Motor-skills learning. Annu. Rev. Psychol, 12:243– 280, 1961. [7] M. Binnard and M. R. Cutkosky. Design by composition for layered manufacturing. Journal of Mechanical Design, 122(1):91–101, 2000. 104
BIBLIOGRAPHY
105
[8] R. Boian, M. Bouzit, G. Burdea, J. Lewis, and J. Deutsch. Dual stewart platform mobility simulator. In 9th International Conference on Rehabilitation Robotics, ICORR, pages 550– 555, June 2005. [9] R. F. Boian, J. E. Deutsch, C. Lee, G. C. Burdea, and J. Lewis. Haptic effects for virtual reality-based post-stroke rehabilitation. In HAPTICS’03, pages 247– 253, 2003. [10] J. Brown. A proposed program of research on psychological feedback(knowledge of results) in the performance of psychomotor tasks. Proc. Res. Plann. Conf. Percept. Mot. Skills, AF HRRC Conf. Rep., 49(2):81–87, 1949. [11] L. Buoguila, M. Ishii, and M. Sato. Multi-modal haptic device for large-scale virtual environments. In MULTIMEDIA ’00: Proceedings of the eighth ACM international conference on Multimedia, pages 277–283, New York, NY, USA, 2000. ACM. [12] C. G. Burgar, P. S. Lum, P. C. Shor, and H. F. M. Van der Loos. Development of robots for rehabilitation therapy: The Palo Alto VA/Stanford experience. Journal of Rehabilitation Research and Development, 37(6):663–673, 2000. [13] M. Casadio, V. Sanguineti, C. Solaro, and P. G. Morasso. A haptic robot reveals the adaptation capability of individuals with multiple sclerosis. Int. J. Rob. Res., 26(11-12):1225–1233, 2007. [14] C. Chia. Low cost virtual cockpits for air combat experimentation. In Interservice/ Industry Training, Simulation, and Education Conference, 2004. [15] K.-S. Choi, H. Sun, and P.-A. Heng. Interactive deformation of soft tissues with haptic feedback for medical learning. Information Technology in Biomedicine, IEEE Transactions, 7:358–363, 2003. [16] R. R. Christensen, J. M. Hollerbach, Y. Xu, and S. G. Meek. Inertial force feedback for the treadport locomotion interface. In Presence: Teleoperators and Virtual Environments, pages 1–14, 2000.
BIBLIOGRAPHY
106
[17] K. Cole and D. Rotella. Old age affects fingertip forces when restraining an unpredictably loaded object. Exp. Brain Res, 136:535–542, 2001. [18] A. Compston and A. Coles. Multiple sclerosis. The Lancet, 359:1221 – 1231, 2002. [19] C. Cruz-Neira, D. J. Sandin, and T. A. DeFanti. Surround-screen projectionbased virtual reality: The design and implementation of the cave. In ACM SIGGRAPH’93, pages 135–142, 1993. [20] F. Danion, M. Descoins, and R. Bootsma. Aging affects the predictive control of grip force during object manipulation. Exp. Brain Res, 2007. [21] R. Darken and W. Banker. Navigating in natural environments: a virtual environment training transfer study. In Virtual Reality Annual Intl. Symposium, IEEE, pages 12–19, March 1998. [22] R. Darken and W. Cockayne. The omni-directional treadmill: A locomotion device for virtual worlds. In UIST ’97, pages 213–221, Banff, Canada, October 1997. [23] S. Davenport. Recovery within grasp. A report on a preliminary study investigating the short and medium term effects of the SaeboFlex (Functional Tone Management System) on chronic post stroke patients with residual upper limb deficit. The Association of Chartered Physiotherapists interested in Neurology, 2005. [24] R. De Veaux, P. Velleman, and D. Bock. Stats Data and Models. AddisonWesley, 2005. [25] K. DG, H. RL, S. S, and R. WZ. Relative contributions of neural mechanisms versu@s muscle mechanics in promoting finger extension deficits following stroke. Muscle Nerve, 28, 2003. [26] N. Durlach and A. Mavor. Virtual reality: Scientific and technological challenges. National Academy Press, 1994.
BIBLIOGRAPHY
107
[27] J. Flanagan, J. Tresilian, and A. Wing. Coupling of grip force and load force during arm movements with grasped objects. Neurosci. Lett., 152:53–56, 1993. [28] J. Freeman, D. Langdon, J. Hobart, and A. Thompson. The impact of inpatient rehabilitation on progressive multiple sclerosis. Ann Neurol, 42:236–244, 1997. [29] E. Garant, A. Daigle, and P. Desbiens. A virtual reality training system for power-utility personnel. In Communications, Computers, and Signal Processing, IEEE Pacific Rim Conference, pages 296–299, May 1995. [30] M. Gilles and A. Wing. Age-related changes in grip force and dynamics of hand movement. J. Mot. Behav, 35:79–85, 2003. [31] R. Goldberg. Psychosocial Aspects of Stroke, in Rehabilitation Psychology. Aspen Publication, 1984. [32] D. E. Goodkin, D. Hertsgaard, and J. Seminary. Upper extremity function in multiple sclerosis: improving assessment sensitivity with box-and-block and nine-hole peg tests. Arch Phys Med Rehabil, 69:850–854, 1988. [33] G. Gresham and P. Duncan. Post-stroke rehabilitation. Clinical practice guideline, volume 16. AHCPR Publication, U.S. Department of Health Services, Washington, DC, 1995. [34] E. Grief and R. Matarazzo. Behavioral Approaches to Rehabilitation: Coping with Change. Springer Publishing Co., 1982. [35] W. B. Griffin, W. R. Provancher, and M. R. Cutkosky. Feedback strategies for telemanipulation with shared control of object handling forces. Presence: Teleoper. Virtual Environ., 14(6):720–731, 2005. [36] N. Hashimoto, S. Jeong, Y. Takeyama, and M. Sato. Immersive multi-projector display on hybrid screens with human-scale haptic and locomotion interfaces. In CW ’04: Proceedings of the 2004 International Conference on Cyberworlds (CW’04), pages 361–368, Washington, DC, USA, 2004. IEEE Computer Society.
BIBLIOGRAPHY
108
[37] V. Hayward and M. Cruz-Hernandez. Tactile display device using distributed lateral skin stretch. In Haptic Interfaces for Virtual Environment and Teleoperator Systems Symposium, pages 1309–1314, Orlando, Florida, USA, 2000. ASME IMECE2000. [38] J. Hermsdorfer, H. E., D. Nowak, and C. Marquardt. Grip force control during object manipulation in cerebral stroke. Clin. Neurophysiol, 114:915–929, 2003. [39] R. L. Hewer.
Rehabilitation after stroke.
Quarterly Journal of Medicine,
76(279):659–674, July 1990. [40] M. Hirose, T. Ogi, H. Yano, and N. Kakehi. Development of wearable force display (hapticgear) for immersive projection displays. In VR ’99: Proceedings of the IEEE Virtual Reality, page 79, Washington, DC, USA, 1999. IEEE Computer Society. [41] L. Holifield. Close-Quarter Combat: A Soldier’s Guide to Hand-To-Hand Fighting. Paladin Press, May 1997. [42] J. Hollerbach. Locomotion interfaces. Handbook of Virtual Environments Technology, pages 239–254, 2002. [43] J. M. Hollerbach, W. B. Thompson, and P. Shirley. The convergence of robotics, vision, and computer graphics for user information. IJRR, 18:1088–1100, 1999. [44] O. Hook. Motivation–neuropsychological views. Lakartidningen, 73(20):1920– 1924, 1976. [45] Immersion.
Cybergrasp
exoskeleton,
immersion
technologies
inc.
http://www.immersion.com/3d/interaction/cybergrasp, 2002. [46] M. Ishii and M. Sato. A 3d spatial interface device using tensed strings. PRESENCE, 3(1):81– 86, 1994. [47] H. Iwata. Locomotion interface for virtual environments. In Robotics research; Proceedings of the 9th International Symposium, Snowbird, UT; UNITED KINGDOM, Oct 1999.
BIBLIOGRAPHY
109
[48] H. Iwata and Y. Yoshida. Path reproduction tests using a torus treadmill. Presence: Teleoper. Virtual Environ., 8(6):587–597, 1999. [49] R. Johansson and G. Westling. Roles of glabrous skin receptors and sensori motor memory in automatic control of precision grip when lifting rougher or more slippery objects. Exp. Brain Res, 56(550-564), 1984. [50] R. S. Johansson and G. Westling. Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects. Experimental Brain Research, 56:550–564, 1984. [51] R. S. Johansson and G. Westling. Signals in tactile afferents from the fingers eliciting adaptive motor responses during precision grip. Experimental Brain Research, 66:141–154, 1987. [52] R. S. Johansson and G. Westling. Tactile afferent input influencing motor coordination during precision grip. Clinical Aspects of Sensory Motor Integration, 1987. [53] R. S. Johansson and G. Westling. Programmed and triggered actions to rapid load changes during precision grip. Experimental Brain Research, 71:72–86, 1988. [54] M. Johnson, H. F. M. Van der Loos, C. Burgar, P. Shor, and L. Leifer. Experimental results using force-feedback cueing in robot-assisted stroke therapy. Neural Systems and Rehabilitation Engineering, IEEE Transactions on, 13, 2005. [55] M. J. Johnson. Embedded Corrective Force Cueing: A Force-Feedback Control Design to Optimize the Motivating Potential of Robot-Assisted Therapy Devices to Increase Bilateral Functioning In Hemiplegic Stroke Patients. PhD thesis, Stanford University, 2001. [56] H. Kawasaki, S. Ito, Y. Ishigure, Y. Nishimoto, T. Aoki, T. Mouri, H. Sakaeda, and M. Abe. Development of a hand motion assist robot for rehabilitation
BIBLIOGRAPHY
110
therapy by patient self-motion control. In Rehabilitation Robotics, 2007. ICORR 2007. IEEE 10th International Conference on, pages 234–240, June 2007. [57] M. Kelly-Hayes and C. Paige. Assessment and psychologic factors in stroke rehabilitation. Neurology, 45(2):S29–32, 1995. [58] R. Klatzky, S. Lederman, and V. Metzger. Identifying objects by touch: An ”expert system”. Perception and Psychohpysics, 37(4):299–302, 1985. [59] KMI.
The hand mentor system.
Kinetic Muscles Inc,
2005.
http://www.kineticmuscles.com. [60] V. Krishnan, P. B. de Freitas, and S. Jaric. Impaired object manipulation in mildly involved individuals with multiple sclerosis. Motor Control, 12:3–20, 2008. [61] A. B. Kuang, S. Payandeh, and B. Zheng. Assembling virtual fixtures for guidance in training environments. In Haptic Interfaces for Virtual Environment and Teleoperator Systems, HAPTICS ’04, 12th Intl. Symposium, pages 367–374, March 2004. [62] J. Lieberman and C. Breazeal. Tikl: Development of a wearable vibrotactile feedback suit for improved human motor learning. IEEE Transactions on Robotics, 23(5):919–926, October, 2007. [63] B. Lowe and A. Freivalds. Effect of carpal tunnel syndrome on grip force coordination on hand tools. Ergonomics, 42:550–564, 1999. [64] M. Macedonia, M. Zyda, D. Pratt, P. Barham, and S. Zeswitz. Npsnet: A network software architecture for large scale virtual environments. Presence, 3(4), 1994. [65] N. Maclean and P. Pound. A critical review of the concept of patient motivation in the literature on physical rehabilitation. Soc Sci Med, 50(4):495–506, 2000.
BIBLIOGRAPHY
111
[66] N. Maclean, P. Pound, and C. Wolfe. Qualitative analysis of stroke patients’ motivation for rehabilitation. British Medical Journal, 321(7268):1051–4, 2000. [67] R. M. Mahoney, H. F. M. Van der Loos, P. S. Lum, and C. Burgar. Robotic stroke therapy assistant. Robotica, 21(1):33–44, 2003. [68] R. Marwaha, S. Hall, C. Knight, and S. Jaric. Load and grip force coordination in static bimanual manipulation tasks in multiple sclerosis. Motor Control, 10:160–177, 2006. [69] E. Moberg. Aspects of sensation in reconstructive surgery of the upper extremity. J. Bone Joint Surg. Am, 46:817–825, 1964. [70] J. Moshell, B. Blau, and B. Knerr. A research testbed for virtual environment training applications. In Virtual Reality Annual Intl. Symposium, IEEE, pages 83–89, Sept 1993. [71] A. M. Murray, R. L. Klatzky, and P. K. Khosla. Psychophysical characterization and testbed validation of a wearable vibrotactile glove for telemanipulation. Presence, 12(2):156–182, 2003. [72] J. Y. C. Ng and J. C. F. Man. Vibro-monitor: A vibrotactile display for physiological data monitoring. In Human Interface Technologies Conference, 2004. [73] D. Nowak, J. Hermsdorfer, S. Glasauer, J. Philipp, L. Meyer, and N. Mai. The effects of digital anaesthesia on predictive grip force adjustments during vertical movements of a grasped object. Eur. J. Neurosci, 14:756–762, 2001. [74] Panasonic.
Panasonic stroke rehabilitation suit.
In Times Mag-
azine (http://www.time.com/time/2006/techguide/bestinventions/inventions/ medicine4.html), 2006. [75] J. Parsons. Fully immersive team training: A networked testbed for groundbased training missions. In Interservice/Industry Training Systems and Education Conference (I/ITSEC), December 1998.
BIBLIOGRAPHY
112
[76] B. Peterson, M. Wells, T. Furness, and E. Hunt. The effects of the interface on navigation in virtual environment. In Proc. Human Factors and Ergonomics Society, pages 1496–1500, Santa Monica, CA, 1998. [77] W. Provancher, M. Cutkosky, K. Kuchenbecker, and G. Niemeyer. Contact location display for haptic perception of curvature and object motion. The International Journal of Robotics Research, 24(9):691–702, 2005. [78] W. R. Provancher, B. D. Erickson, F. Barbagli, and H. Z. Tan. Tactile perception of rotational sliding. In WHC ’07: Proceedings of the Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pages 127–132, Washington, DC, USA, 2007. IEEE Computer Society. [79] K. T. Ragnarsson. Functional electrical stimulation after spinal cord injury: current use, therapeutic effects and future directions. Spinal Cord, 46, 2008. [80] J. Reece. Virtual close quarter battle(cqb) graphic decision trainer. In Master thesis, Monterey, CA, 2002. Naval Postgraduate School. [81] D. M. E. Robert T. Fraser, George H. Kraft and K. L. Johnson. The MS Workbook: Living Fully with Multiple Sclerosis. New Harbinger Publications, 2006. [82] M. Rocca, B. Colombo, A. Falini, A. Ghezzi, V. Martinelli, G. Scotti, G. Comi, and M. Filippi. Cortical adaptation in patients with ms: a cross-sectional functional mri-study of diseases phenotypes. Lancet Neurol, 4:618–626, 2005. [83] F. D. Rose. Training in virtual environments: Transfer to real world tasks and equivalence to real task training. Lancet Neurol, 43:494–511, 2000. [84] G. Roston and T. Peurach. A whole body kinesthetic display device for virtual reality applications. In Proceedings of IEEE International Conference on Robotics and Automation, pages 3006–3011, Albuquerque, New Mexico, 1997.
BIBLIOGRAPHY
113
[85] R. A. Rudick, G. Cutter, and S. Reingold. The multiple sclerosis functional composite a new clinical outcome measure for multiple sclerosis trials. Multiple Sclerosis, 8:359–365, 2002. [86] H. Schmidt, S. Hesse, R. Bernhardt, and J. Kr¨ uger. Hapticwalker—a novel haptic foot device. ACM Trans. Appl. Percept., 2(2):166–180, 2005. [87] R. A. Schmidt. Motor Control and Learning. Human Kinetics Publishers, 1982. [88] R. Shadmehr and F. Mussa-Ivaldi. Adaptive representation of dynamics during learning of a motor task. Journal of Neuroscience, 14(5 Pt 2), 1994. [89] M. Shinohara, M. Latash, and V. Zatsiorsky. Age effects on force produced by intrinsic and extrinsic hand muscles and finger interaction during mvc tasks. J. Appl. Physiol, 95(1361-1369), 2003. [90] M. Slater, A. Steed, and M. Usoh.
The virtual treadmill: A naturalistic
metaphor for navigation in immersive virtual environments. In Virtual Environments’ 95, pages 135–148. Springer-Verlag, 1995. [91] J. Solis, C. Avizzano, and M. Bergamasco. Validating a skill transfer system based on reactive robots technology. In Robot and Human Interactive Communication, ROMAN 2003. The 12th IEEE Intl, October 2003. [92] J. E. Speich, L. Shao, and M. Goldfarb. Modeling the human hand as it interacts with a telemanipulation system. Mechatronics,, 15(9):1127–1142, 2005. [93] E. Taub, J. E. Crago, and G. Uswatte. Constraint-induced movement therapy: A new approach to treatment in physical rehabilitation. Rehab. Psych., 42(2), 1998. [94] E. Taub, G. Uswatte, and R. Pidikiti. Constraint-induced movement therapy: A new family of techniques with broad application to physical rehabilitationa clinical review. J. Rehab. Res. Devel., 36(3), 1999.
BIBLIOGRAPHY
114
[95] E. Taub and S. L. Wolf. Constraint-induced movement techniques to facilitate upper extremity use in stroke patients. Top. Stroke Rehab., 3, 1997. [96] S. Thompson, A. Sobolew-Shubin, and M. Graham. Psychosocial adjustment following a stroke. Soc Sci Med, 28(3):239–47, 1989. [97] A. Toney, L. Dunne, B. H. Thomas, and S. P. Ashdown. A shoulder pad insert vibrotactile display. In ISWC ’03: Proceedings of the 7th IEEE International Symposium on Wearable Computers, page 35, Washington, DC, USA, 2003. IEEE Computer Society. [98] D. Traum, J. Rickel, J. Gratch, and S. Marsella. Negotiation over tasks in hybrid human-agent teams for simulation-based training. In AAMAS ’03: Proceedings of the second international joint conference on Autonomous agents and multiagent systems, pages 441–448, New York, NY, USA, 2003. ACM. [99] K. Tsukada and M. Yasumura. Activebelt: Belt-type wearable tactile display for directional navigation. Book Series Lecture Notes in Computer Science, pages 384–399, 2004. [100] H. F. M. Van der Loos. VA/Stanford rehabilitation robotics research and development program: Lessons learned in the application. of robotics technology to the field of rehabilitation. Rehabilitation Engineering, IEEE Transactions on, 3:46–55, Mar 1995. [101] D. Waller, E. Hunt, and D. Knapp. The transfer of spatial knowledge in virtual environment training. resence: Teleoperators and Virtual Environments, 7(2):129–143, 1998. [102] M. Ward, R. Azuma, R. Bennett, S. Gottschalk, and H. Fuchs. A demonstrated optical tracker with scalable work area for head-mounted display system. In Symp. Interactive 3-D Graphics, pages 43–52, 1992. [103] L. E. Weiss, R. Merz, F. B. Prinz, G. Neplotnik, P. Padmanabhan, L. Schultz, and K. Ramaswami. Shape deposition manufacturing of heterogenous structures. Journal of Manufacturing Systems, 16(4):239–248, 1997.
BIBLIOGRAPHY
115
[104] Westling.G and R. Johansson. Responses in glabrous skin mechanoreceptors during precision grip in humans. Exp. Brain Res, 66:128–140, 1987. [105] B. Witmer, J. Bailey, and B. Knerr. Training dismounted soldiers in virtual environments: Route learning and transfer. U.S. Army Research Institute for the Behavioral and Social Sciences, 1995. [106] A. Yamamoto, B. Cros, H. Hashimoto, and T. Higuchi. Control of thermal tactile display based on prediction of contact temperature. In Robotics and Automation, 2004. Proceedings. ICRA ’04. 2004 IEEE International Conference, volume 2, pages 1536–1541, 2004. [107] D. Zeltzer and N. J. Pioch. Validation and verification of virtual environment training systems. In Virtual Reality Annual Intl. Symposium, IEEE, pages 123– 130, March 1996. [108] J. Zwislocki and D. Goodman. Absolute scaling of sensory magnitudes: a validation. Percept Psychophys, 28(1):28–38, 1980.
![IM [PROVE] SCALE. Training Course Feedback Form](https://kipdf.com/img/300x300/im-prove-scale-training-course-feedback-form_5ad784b77f8b9a5f248b45f3.jpg)