machine learning for hand prosthetics (and more)

machine learning for hand prosthetics (and more) barbara CAPUTO1, claudio CASTELLINI2, gerd HIRZINGER3, luo JIE1, giorgio METTA2,4, francesco ORABONA...
2 downloads 0 Views 905KB Size
machine learning for hand prosthetics (and more)

barbara CAPUTO1, claudio CASTELLINI2, gerd HIRZINGER3, luo JIE1, giorgio METTA2,4, francesco ORABONA1, giulio SANDINI2,4, patrick VAN DER SMAGT3

1

IDIAP, martigny, switzerland LIRA-Lab, university of genova, italy 3 German Aerospace Research Center, oberpfaffenhofen, germany 4 Italian Institute of Technology, genova, italy 2

www.neurobotics.org  robotic artifacts driven by humans (HBS), e.g.  intelligent prostheses  intelligent teleoperation platforms

 …where by intelligent we mean semiautonomous and/or adaptive  solution: learn models of complex actions, e.g., reaching, grasping, etc. that can be used by HBSs autumn ‘07

ml for hand prosthetics (and more)

2

www.neurobotics.org  in intelligent prosthetics:  guess what the patient wants his prosthesis to do

 in teleoperated reaching and grasping:  guess when the master wants to grasp  guess how the master wants to grasp  guess what the master wants to grasp

autumn ‘07

ml for hand prosthetics (and more)

3

outline  application of machine learning to  an emg-controlled hand prosthesis  indoor robotic navigation  human reaching and grasping

 some optimisations (on the road)  uniform sampling in the input space  linear independence in the feature space

autumn ‘07

ml for hand prosthetics (and more)

4

emg-driven hand prosthesis

autumn ‘07

ml for hand prosthetics (and more)

5

problem  high dexterity, but  little possibility of control by the patient: what interface?  we focus upon non-invasive interfaces, particularly upon  forearm surface electromyography  can a mechanical hand be swiftly driven using the emg? autumn ‘07

ml for hand prosthetics (and more)

6

setup

    

10 Ottobock emg electrodes 1 SpaceControl force/torque sensor 4 fingertip sensors detect type of grasps detect force involved in the grasp

autumn ‘07

ml for hand prosthetics (and more)

7

emg (1)

 in principle, non-linearly related to the force applied by a muscle autumn ‘07

ml for hand prosthetics (and more)

8

emg (2)  a very badly conditioned signal  affected by a number of factors: 1. (long-term) inter-arm differences 2. (long-term) arm postures and movements 3. (short-term) muscular fatigue, sweat 4. (medium-term) electrode displacement, muscle cross-talk

 how to take into account all these problems? autumn ‘07

ml for hand prosthetics (and more)

9

emg (3)  how to take into account all these problems? 1. (neglected) one subject only, able-bodied 2. (neglected) relaxed on a table, fixed position 3. three-and-a-half minutes of activity (one session) 4. three sessions form a group; electrodes replaced between groups

 all in all, 30 sessions in 10 groups across 2 days autumn ‘07

ml for hand prosthetics (and more)

10

methods  feed-forward neural network (classification, regression)  sigmoidal activation  one hidden layer w/10 neurons  backprop

 support vector machine (classification, regression)  gaussian kernel

 locally weighted projection regression (regression only)  online, incremental autumn ‘07

ml for hand prosthetics (and more)

11

svc cross-session accuracy sessions 1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

100

1 2 90

3 4

80

5 6

70

Aij =

models

7 8 9

60

10 11

50

12 13

40

14 15 30

Aij is the classification accuracy of model trained upon session i, tested on session j. graphs refer to day 1 – day 2 is analogous. autumn ‘07

ml for hand prosthetics (and more)

12

uniformisation sessions

sessions 1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

100

3

4

5

6

7

8

9

10

11

12

13

14

15

100

1

1

2

2 90

3

90

3 4

4 80

5

80

5 6

6 70

8 9

60

70

7

models

7

models

2

8 9

60

10

10 11

50

11

50

12

12 13

40

13

40

14

14

15

15

30

30

full training sets

uniform training sets

diag: 98.73% ± 0.39% non-diag: 73.23% ± 14.29%

diag: 95.52% ± 1.21% non-diag: 74.53% ± 13.70%

autumn ‘07

ml for hand prosthetics (and more)

13

what makes it hard?  average minimum sample set distance:

 Dij is strongly correlated to Aij (Pearson coefficient: -0.60)  analogous results with nn and on regression (all approaches)  what if we adjoin good models and then train on this new model? autumn ‘07

ml for hand prosthetics (and more)

14

best models 30

100

Support Vector Machine Neural Network LWPR

95

25

90

20

error (%)

accuracy (%)

Support Vector Machine Neural Network

85

15

80

10

75

5

70

1

2

3

4

5

6

7

8 9 sessions

10

11

12

13

14

15

0

1

2

3

4

5

6

7

8 9 sessions

10

11

12

13

14

15

best models on day 1, classification accuracy (left) and regression NRMSE (right)

autumn ‘07

ml for hand prosthetics (and more)

15

intermission (1)  information is there, one only has to dig and find it!  distance is the key  need a mechanical way of building good models  on-line version of uniformisation:  check wheter a new sample is far away from the current training set  if so, use it  otherwise, ignore it autumn ‘07

ml for hand prosthetics (and more)

16

growth of a training set size (# of samples)

2000

growth is expected to come to a halt eventually...

1500 1000 500 0

0

1000

2000

3000

4000

5000

size (% of total)

5 4 3 2 1

autumn ‘07

500

1000

1500

2000 2500 3000 time (secs)

3500

4000

4500

ml for hand prosthetics (and more)

5000

17

on-line accuracy 100

black line says: point by point, we’re finding what you need. red line says: we aren’t going to miss anything.

95 90

accuracy (%)

85 80 75 70 65 60 growing testing set full testing set

55 50

0

autumn ‘07

10

20

30 training step

40

ml for hand prosthetics (and more)

50

18

as the distance grows... (1) 10000 9000 8000

training set size

7000

training set size decreases as d-10

6000 5000 4000 3000 2000 1000 0

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

d autumn ‘07

ml for hand prosthetics (and more)

19

as the distance grows... (2) 100 Support Vector Machine Neural Network

95 90

accuracy (%)

85

accuracy decreases linearly.

80 75 70 65 60 55 50

autumn ‘07

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

ml for hand prosthetics (and more)

20

as the distance grows... (3) 20 Support Vector Machine Neural Network LWPR

NRMSE (%)

15

10

5

0.15

0.2

0.25

0.3

0.35

0.4

0.45

error increases linearly. for d=0.53, error is 12.12%, and the training set has 77 samples over 153.000!

0.5

d autumn ‘07

ml for hand prosthetics (and more)

21

conclusions  problem is solved so far!  now, gather more data to take arm movement into account  the online uniformisation seems a promising way to keep training sets small and effective

 then, control the hand, and lastly...  ...try it on a patient.

autumn ‘07

ml for hand prosthetics (and more)

22

learning to recognise places  two mobile robots gather images of a complex building in different conditions of lighting, camera position, settings  they are then asked to recognise the places they’ve been to

autumn ‘07

ml for hand prosthetics (and more)

23

learning to recognise places 450

7000

OISVM LIBSVM−2 LIBSVM

400

OISVM, η=0.10 LIBSVM−2 LIBSVM 6000

350

Number of Support Vectors

Number of Support Vectors

5000

300

250

200

150

4000

3000

2000

100 1000

50

0

0

0

100

200

300 Number of samples

400

500

600

0

1000

2000

3000

4000

5000 6000 7000 Number of samples

8000

9000

10000

11000

growth: the size of the models reaches a maximum value and then stops (left pane, finite kernel) or it grows much less (right pane, infinite kernel). the accuracy of the machines is exactly the same. autumn ‘07

ml for hand prosthetics (and more)

24

learning to recognise places 95

900 800

90 Number of Support Vectors

Classification Rate (%)

700

85

80

75 Fixed−partition Memory−controlled OISVM, η=0.01 OISVM, η=0.017 OISVM, η=0.025

70

65 0

2

4

6

8 10 Training Step

12

14

Fixed−partition Memory−controlled OISVM, η=0.01 OISVM, η=0.017 OISVM, η=0.025

600 500 400 300 200 100

16

0 0

2

4

6

8 10 Training Step

12

14

16

accuracy: one can obtain massively smaller models, if accuracy is allwed to degrade by a maximum of 4%.

autumn ‘07

ml for hand prosthetics (and more)

25

learning more efficiently  OISVM: Online Independent Support Vector Machines keep models small by exploiting linear independence in the feature space:  check whether a new sample is  if so, use it  otherwise, ignore it.

 suited for online learning, where  data appear out-of-the-blue with no possible reordering and/or statistics  data flow is potentially endless

 no approximation involved, or able to tune the approximation autumn ‘07

ml for hand prosthetics (and more)

26

what is l.i. in the feature space?  recall your I year linear algebra:

 in real life, you need to check whether ∆ is less than some η (linearly dependent) or greater than it (linearly independent) autumn ‘07

ml for hand prosthetics (and more)

27

what is l.i. in the feature space?

 in general, ignore xl+1 if K-close to xi  if K is gaussian, then ignore xl+1 if close to xi... does this ring a bell? autumn ‘07

ml for hand prosthetics (and more)

28

intermission (2) “When computers are applied to solve a practical problem it is usually the case that the method of deriving the required output from a set of inputs can be described explicitly. […] As computers are applied to solve more and more complex tasks, however, situations can arise when there is no known method for computing the desired output from a set of inputs, or where that computation may be very expensive.”

[Cristianini & Shawe-Taylor, An introduction to Support Vector Machines, 2000] autumn ‘07

ml for hand prosthetics (and more)

29

learning when to grasp  human subjects look at objects on a monitor and simulate the act of grasping  by monitoring the user’s gaze and arm motion, guess when he wants to grasp  consider an adequate time window of gaze variance and arm velocity

autumn ‘07

ml for hand prosthetics (and more)

30

learning when to grasp 100

accuracy (%)

95

90

85

80 SVM thresholds 75 0

1

2

3

4

5

T

c

comparison: Support Vector Machines vs. a simple decision tree autumn ‘07

ml for hand prosthetics (and more)

31

learning when to grasp

accuracy (%)

100

95 1 2 3 4 5 6 7

90

85 0

1

2

3

4

5

T

c

accuracy: how well do we guess the subject’s intention? autumn ‘07

ml for hand prosthetics (and more)

32

learning when to grasp 70

1 2 3 4 5 6 7

pSV (model size)

60 50 40 30 20 10 0 85

90

95

100

accuracy (%)

compactness: how large are the models obtained? autumn ‘07

ml for hand prosthetics (and more)

33

learning when to grasp SVMs build excellent models of the user’s intention to grasp:  accurate  small (i.e., fast and suitable for online applications)

the problem is easy in the feature space, not necessarily in the input space! research about (functional) biological plausibility is ongoing autumn ‘07

ml for hand prosthetics (and more)

34

learning how to grasp

autumn ‘07

ml for hand prosthetics (and more)

35

learning how to grasp 160 150

thumb rot thumb inn thumb out

140

degrees

130 120 110 100 90 80 70 0

0.1

0.2

0.3

0.4 time

0.5

0.6

0.7

0.78

0

2

4

6

8

10

time

accuracy: how well do we guess the subject’s grasping style? show the system the beginning of the trajectories (22 joints + 6 position/orientation) and figure out the final position autumn ‘07

ml for hand prosthetics (and more)

36

learning how to grasp  allows for a precise prediction of  hand position: 0.5in  hand orientation: 2.5°  hand posture: 7.5°

 with a reasonable advance:  hand position: 200msecs.  hand orientation: 120msecs.  hand posture: 90-200msecs. (depending on the object) autumn ‘07

ml for hand prosthetics (and more)

37

learning how to grasp hand orientation

hand posture

7

14 13

6 12 11

error (degrees)

error (degrees)

5

4

3

9 8 7 6

2 overall single object 1 0.1

10

0.15

0.2

0.25

0.3 B

0.35

0.4

0.45

overall single object

5

0.5

4 0.1

0.15

0.2

0.25

0.3 B

0.35

0.4

0.45

0.5

affordance: knowing a priori the object will uniformly improve the error, at no additional computational overhead.

autumn ‘07

ml for hand prosthetics (and more)

38

conclusions  ml will teach your machines useful concepts such as human grasping, reaching, intentions  those models can be trasferred to humanoid platforms (but not only to them)  they can be employed in aiding the disabled  lots of lovely maths behind it! autumn ‘07

ml for hand prosthetics (and more)

39

thank you!