machine learning for hand prosthetics (and more)
barbara CAPUTO1, claudio CASTELLINI2, gerd HIRZINGER3, luo JIE1, giorgio METTA2,4, francesco ORABONA1, giulio SANDINI2,4, patrick VAN DER SMAGT3
1
IDIAP, martigny, switzerland LIRA-Lab, university of genova, italy 3 German Aerospace Research Center, oberpfaffenhofen, germany 4 Italian Institute of Technology, genova, italy 2
www.neurobotics.org robotic artifacts driven by humans (HBS), e.g. intelligent prostheses intelligent teleoperation platforms
…where by intelligent we mean semiautonomous and/or adaptive solution: learn models of complex actions, e.g., reaching, grasping, etc. that can be used by HBSs autumn ‘07
ml for hand prosthetics (and more)
2
www.neurobotics.org in intelligent prosthetics: guess what the patient wants his prosthesis to do
in teleoperated reaching and grasping: guess when the master wants to grasp guess how the master wants to grasp guess what the master wants to grasp
autumn ‘07
ml for hand prosthetics (and more)
3
outline application of machine learning to an emg-controlled hand prosthesis indoor robotic navigation human reaching and grasping
some optimisations (on the road) uniform sampling in the input space linear independence in the feature space
autumn ‘07
ml for hand prosthetics (and more)
4
emg-driven hand prosthesis
autumn ‘07
ml for hand prosthetics (and more)
5
problem high dexterity, but little possibility of control by the patient: what interface? we focus upon non-invasive interfaces, particularly upon forearm surface electromyography can a mechanical hand be swiftly driven using the emg? autumn ‘07
ml for hand prosthetics (and more)
6
setup
10 Ottobock emg electrodes 1 SpaceControl force/torque sensor 4 fingertip sensors detect type of grasps detect force involved in the grasp
autumn ‘07
ml for hand prosthetics (and more)
7
emg (1)
in principle, non-linearly related to the force applied by a muscle autumn ‘07
ml for hand prosthetics (and more)
8
emg (2) a very badly conditioned signal affected by a number of factors: 1. (long-term) inter-arm differences 2. (long-term) arm postures and movements 3. (short-term) muscular fatigue, sweat 4. (medium-term) electrode displacement, muscle cross-talk
how to take into account all these problems? autumn ‘07
ml for hand prosthetics (and more)
9
emg (3) how to take into account all these problems? 1. (neglected) one subject only, able-bodied 2. (neglected) relaxed on a table, fixed position 3. three-and-a-half minutes of activity (one session) 4. three sessions form a group; electrodes replaced between groups
all in all, 30 sessions in 10 groups across 2 days autumn ‘07
ml for hand prosthetics (and more)
10
methods feed-forward neural network (classification, regression) sigmoidal activation one hidden layer w/10 neurons backprop
support vector machine (classification, regression) gaussian kernel
locally weighted projection regression (regression only) online, incremental autumn ‘07
ml for hand prosthetics (and more)
11
svc cross-session accuracy sessions 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
100
1 2 90
3 4
80
5 6
70
Aij =
models
7 8 9
60
10 11
50
12 13
40
14 15 30
Aij is the classification accuracy of model trained upon session i, tested on session j. graphs refer to day 1 – day 2 is analogous. autumn ‘07
ml for hand prosthetics (and more)
12
uniformisation sessions
sessions 1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
1
100
3
4
5
6
7
8
9
10
11
12
13
14
15
100
1
1
2
2 90
3
90
3 4
4 80
5
80
5 6
6 70
8 9
60
70
7
models
7
models
2
8 9
60
10
10 11
50
11
50
12
12 13
40
13
40
14
14
15
15
30
30
full training sets
uniform training sets
diag: 98.73% ± 0.39% non-diag: 73.23% ± 14.29%
diag: 95.52% ± 1.21% non-diag: 74.53% ± 13.70%
autumn ‘07
ml for hand prosthetics (and more)
13
what makes it hard? average minimum sample set distance:
Dij is strongly correlated to Aij (Pearson coefficient: -0.60) analogous results with nn and on regression (all approaches) what if we adjoin good models and then train on this new model? autumn ‘07
ml for hand prosthetics (and more)
14
best models 30
100
Support Vector Machine Neural Network LWPR
95
25
90
20
error (%)
accuracy (%)
Support Vector Machine Neural Network
85
15
80
10
75
5
70
1
2
3
4
5
6
7
8 9 sessions
10
11
12
13
14
15
0
1
2
3
4
5
6
7
8 9 sessions
10
11
12
13
14
15
best models on day 1, classification accuracy (left) and regression NRMSE (right)
autumn ‘07
ml for hand prosthetics (and more)
15
intermission (1) information is there, one only has to dig and find it! distance is the key need a mechanical way of building good models on-line version of uniformisation: check wheter a new sample is far away from the current training set if so, use it otherwise, ignore it autumn ‘07
ml for hand prosthetics (and more)
16
growth of a training set size (# of samples)
2000
growth is expected to come to a halt eventually...
1500 1000 500 0
0
1000
2000
3000
4000
5000
size (% of total)
5 4 3 2 1
autumn ‘07
500
1000
1500
2000 2500 3000 time (secs)
3500
4000
4500
ml for hand prosthetics (and more)
5000
17
on-line accuracy 100
black line says: point by point, we’re finding what you need. red line says: we aren’t going to miss anything.
95 90
accuracy (%)
85 80 75 70 65 60 growing testing set full testing set
55 50
0
autumn ‘07
10
20
30 training step
40
ml for hand prosthetics (and more)
50
18
as the distance grows... (1) 10000 9000 8000
training set size
7000
training set size decreases as d-10
6000 5000 4000 3000 2000 1000 0
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
d autumn ‘07
ml for hand prosthetics (and more)
19
as the distance grows... (2) 100 Support Vector Machine Neural Network
95 90
accuracy (%)
85
accuracy decreases linearly.
80 75 70 65 60 55 50
autumn ‘07
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
ml for hand prosthetics (and more)
20
as the distance grows... (3) 20 Support Vector Machine Neural Network LWPR
NRMSE (%)
15
10
5
0.15
0.2
0.25
0.3
0.35
0.4
0.45
error increases linearly. for d=0.53, error is 12.12%, and the training set has 77 samples over 153.000!
0.5
d autumn ‘07
ml for hand prosthetics (and more)
21
conclusions problem is solved so far! now, gather more data to take arm movement into account the online uniformisation seems a promising way to keep training sets small and effective
then, control the hand, and lastly... ...try it on a patient.
autumn ‘07
ml for hand prosthetics (and more)
22
learning to recognise places two mobile robots gather images of a complex building in different conditions of lighting, camera position, settings they are then asked to recognise the places they’ve been to
autumn ‘07
ml for hand prosthetics (and more)
23
learning to recognise places 450
7000
OISVM LIBSVM−2 LIBSVM
400
OISVM, η=0.10 LIBSVM−2 LIBSVM 6000
350
Number of Support Vectors
Number of Support Vectors
5000
300
250
200
150
4000
3000
2000
100 1000
50
0
0
0
100
200
300 Number of samples
400
500
600
0
1000
2000
3000
4000
5000 6000 7000 Number of samples
8000
9000
10000
11000
growth: the size of the models reaches a maximum value and then stops (left pane, finite kernel) or it grows much less (right pane, infinite kernel). the accuracy of the machines is exactly the same. autumn ‘07
ml for hand prosthetics (and more)
24
learning to recognise places 95
900 800
90 Number of Support Vectors
Classification Rate (%)
700
85
80
75 Fixed−partition Memory−controlled OISVM, η=0.01 OISVM, η=0.017 OISVM, η=0.025
70
65 0
2
4
6
8 10 Training Step
12
14
Fixed−partition Memory−controlled OISVM, η=0.01 OISVM, η=0.017 OISVM, η=0.025
600 500 400 300 200 100
16
0 0
2
4
6
8 10 Training Step
12
14
16
accuracy: one can obtain massively smaller models, if accuracy is allwed to degrade by a maximum of 4%.
autumn ‘07
ml for hand prosthetics (and more)
25
learning more efficiently OISVM: Online Independent Support Vector Machines keep models small by exploiting linear independence in the feature space: check whether a new sample is if so, use it otherwise, ignore it.
suited for online learning, where data appear out-of-the-blue with no possible reordering and/or statistics data flow is potentially endless
no approximation involved, or able to tune the approximation autumn ‘07
ml for hand prosthetics (and more)
26
what is l.i. in the feature space? recall your I year linear algebra:
in real life, you need to check whether ∆ is less than some η (linearly dependent) or greater than it (linearly independent) autumn ‘07
ml for hand prosthetics (and more)
27
what is l.i. in the feature space?
in general, ignore xl+1 if K-close to xi if K is gaussian, then ignore xl+1 if close to xi... does this ring a bell? autumn ‘07
ml for hand prosthetics (and more)
28
intermission (2) “When computers are applied to solve a practical problem it is usually the case that the method of deriving the required output from a set of inputs can be described explicitly. […] As computers are applied to solve more and more complex tasks, however, situations can arise when there is no known method for computing the desired output from a set of inputs, or where that computation may be very expensive.”
[Cristianini & Shawe-Taylor, An introduction to Support Vector Machines, 2000] autumn ‘07
ml for hand prosthetics (and more)
29
learning when to grasp human subjects look at objects on a monitor and simulate the act of grasping by monitoring the user’s gaze and arm motion, guess when he wants to grasp consider an adequate time window of gaze variance and arm velocity
autumn ‘07
ml for hand prosthetics (and more)
30
learning when to grasp 100
accuracy (%)
95
90
85
80 SVM thresholds 75 0
1
2
3
4
5
T
c
comparison: Support Vector Machines vs. a simple decision tree autumn ‘07
ml for hand prosthetics (and more)
31
learning when to grasp
accuracy (%)
100
95 1 2 3 4 5 6 7
90
85 0
1
2
3
4
5
T
c
accuracy: how well do we guess the subject’s intention? autumn ‘07
ml for hand prosthetics (and more)
32
learning when to grasp 70
1 2 3 4 5 6 7
pSV (model size)
60 50 40 30 20 10 0 85
90
95
100
accuracy (%)
compactness: how large are the models obtained? autumn ‘07
ml for hand prosthetics (and more)
33
learning when to grasp SVMs build excellent models of the user’s intention to grasp: accurate small (i.e., fast and suitable for online applications)
the problem is easy in the feature space, not necessarily in the input space! research about (functional) biological plausibility is ongoing autumn ‘07
ml for hand prosthetics (and more)
34
learning how to grasp
autumn ‘07
ml for hand prosthetics (and more)
35
learning how to grasp 160 150
thumb rot thumb inn thumb out
140
degrees
130 120 110 100 90 80 70 0
0.1
0.2
0.3
0.4 time
0.5
0.6
0.7
0.78
0
2
4
6
8
10
time
accuracy: how well do we guess the subject’s grasping style? show the system the beginning of the trajectories (22 joints + 6 position/orientation) and figure out the final position autumn ‘07
ml for hand prosthetics (and more)
36
learning how to grasp allows for a precise prediction of hand position: 0.5in hand orientation: 2.5° hand posture: 7.5°
with a reasonable advance: hand position: 200msecs. hand orientation: 120msecs. hand posture: 90-200msecs. (depending on the object) autumn ‘07
ml for hand prosthetics (and more)
37
learning how to grasp hand orientation
hand posture
7
14 13
6 12 11
error (degrees)
error (degrees)
5
4
3
9 8 7 6
2 overall single object 1 0.1
10
0.15
0.2
0.25
0.3 B
0.35
0.4
0.45
overall single object
5
0.5
4 0.1
0.15
0.2
0.25
0.3 B
0.35
0.4
0.45
0.5
affordance: knowing a priori the object will uniformly improve the error, at no additional computational overhead.
autumn ‘07
ml for hand prosthetics (and more)
38
conclusions ml will teach your machines useful concepts such as human grasping, reaching, intentions those models can be trasferred to humanoid platforms (but not only to them) they can be employed in aiding the disabled lots of lovely maths behind it! autumn ‘07
ml for hand prosthetics (and more)
39
thank you!