Motion Capture Specialized Motion Capture N. Alberto Borghese Laboratory of Human Motion Analysis and Virtual Reality (MAVR) Department of Computer Science University of Milano
Laboratory of Motion Analysis & Virtual Reality, MAVR
1/49
http://homes.dsi.unimi .it/~borghese/
Outline Introduction: what is Motion Capture? History and Motion Capture technologies. Passive Markers Motion Capture. Specialized motion capture: hand, gaze and face. From Motion Capture to Animation (post-processing)
Laboratory of Motion Analysis & Virtual Reality, MAVR
2/49
http://homes.dsi.unimi .it/~borghese/
1
Gloves Monitor fingers position and force. Problems with the motion of the fingers: • overlap. • fine movements. • fast movements. • rich repertoire.
Laboratory of Motion Analysis & Virtual Reality, MAVR
3/49
http://homes.dsi.unimi .it/~borghese/
Sayre glove (1976)
Laboratory of Motion Analysis & Virtual Reality, MAVR
4/49
http://homes.dsi.unimi .it/~borghese/
2
MIT glove (1977)
5/49
http://homes.dsi.unimi .it/~borghese/
6/49
http://homes.dsi.unimi .it/~borghese/
Digital Data Entry Glove (1983)
Laboratory of Motion Analysis & Virtual Reality, MAVR
Laboratory of Motion Analysis & Virtual Reality, MAVR
3
Data Glove (1987)
Laboratory of Motion Analysis & Virtual Reality, MAVR
7/49
http://homes.dsi.unimi .it/~borghese/
Power Glove (1990)
Laboratory of Motion Analysis & Virtual Reality, MAVR
8/49
http://homes.dsi.unimi .it/~borghese/
4
Cyber Glove (1995)
Laboratory of Motion Analysis & Virtual Reality, MAVR
9/49
http://homes.dsi.unimi .it/~borghese/
Calibration Estimate of the geometrical parameters in the transformation operated by the sensors (e.g. the perspective transformation operated by a video-camera). Estimate of the parameters, which describe distortions introduced by the measurement system. Measurement of a known pattern. From its distortion, the parameters can be computed. Algorithms adopted: polynomial, local correction (neural networks, fuzzy).
Laboratory of Motion Analysis & Virtual Reality, MAVR
10/49
http://homes.dsi.unimi .it/~borghese/
5
Haptic displays Convey to the subject the sensorial information generated in the interaction with the virtual objects: force, material texture… Measure the force exerted by the subject on the virtual environment. Aptic displays provide a mechanical interface for Virtual Reality applications. Most important developments have been made in the robotics field.
Laboratory of Motion Analysis & Virtual Reality, MAVR
11/49
http://homes.dsi.unimi .it/~borghese/
Requirements of haptic displays • Large bandwidth. • Low intertial and viscosity.
Technological solutions : • Direct drive manipulandum (Yoshikawa, 1990), Phantom (2000). • Parallel manipulandum (Millman and Colgate, 1991; Buttolo and Hannaford, 1995). • Magnetic levitation devices (Salcudean and Yan, 1994; Gomi and Kawato, 1996). • Gloves (Bergamasco, 1993).
Laboratory of Motion Analysis & Virtual Reality, MAVR
12/49
http://homes.dsi.unimi .it/~borghese/
6
Direct drive manipulandum (phantom)
Laboratory of Motion Analysis & Virtual Reality, MAVR
13/49
http://homes.dsi.unimi .it/~borghese/
Parallel manipulandum (schema) Hannaford et al.
Laboratory of Motion Analysis & Virtual Reality, MAVR
14/49
http://homes.dsi.unimi .it/~borghese/
7
Pen haptic display
Laboratory of Motion Analysis & Virtual Reality, MAVR
15/49
http://homes.dsi.unimi .it/~borghese/
Gloves (Gini et al., Blackfinger, 2000)
Laboratory of Motion Analysis & Virtual Reality, MAVR
16/49
http://homes.dsi.unimi .it/~borghese/
8
Percro gloves (Begamasco, 1993)
Laboratory of Motion Analysis & Virtual Reality, MAVR
17/49
http://homes.dsi.unimi .it/~borghese/
Gaze input •Contact lenses carrying magnetic coils. •Tvcameras aligned with an IR LED source. •Stereoscopic eye-wear. • The direction of gaze is decided by measuring the shape of the spot reflected by the frontal portion of the cornea (Ohshima et al., 1996).
Laboratory of Motion Analysis & Virtual Reality, MAVR
18/49
http://homes.dsi.unimi .it/~borghese/
9
Outline Introduction: what is Motion Capture? History and Motion Capture technologies. Passive Markers Motion Capture. Specialized motion capture: hand, gaze and face. From Motion Capture to Animation (post-processing).
Laboratory of Motion Analysis & Virtual Reality, MAVR
19/49
http://homes.dsi.unimi .it/~borghese/
Maria Callas: Virtual Tosca
Laboratory of Motion Analysis & Virtual Reality, MAVR
20/49
http://homes.dsi.unimi .it/~borghese/
10
Performance-driven Animation based on the motion capture (in some cases, in realtime) of an actor. Types of performance-driven: •Expression mapping •Model-based persona transmission
Laboratory of Motion Analysis & Virtual Reality, MAVR
21/49
http://homes.dsi.unimi .it/~borghese/
Expression mapping •Images of 20 expressions. •Identify the correspondance between the image and the character in neutral position. •Computation of the deformation field for the character. •Application of the deformation field to the character (possibility of exaggerating the expression). •Tony de Peltrie, 1985.
Laboratory of Motion Analysis & Virtual Reality, MAVR
22/49
http://homes.dsi.unimi .it/~borghese/
11
Model-based Persona Transmission, feature based Identifying the features to map the model to the character.
Laboratory of Motion Analysis & Virtual Reality, MAVR
23/49
http://homes.dsi.unimi .it/~borghese/
Model-based Persona Transmission, mesh based
•Deformation of a topological mesh induced by a control mesh. •The control mesh connects the marker points. Laboratory of Motion Analysis & Virtual Reality, MAVR
24/49
http://homes.dsi.unimi .it/~borghese/
12
Markers disposition
Position of the feature points according to MPEG-4 standard: u principali l secondari
Laboratory of Motion Analysis & Virtual Reality, MAVR
Problems with: Eyes and tongue. Nose basis (visibility).
25/49
http://homes.dsi.unimi .it/~borghese/
Construction of the Control Mesh
47 markers on the skin: - Problems with: Eyes and tongue. Nose basis (visibility). 4 markers on an elastic band: To identify a local Reference Frame (LRF). Laboratory of Motion Analysis & Virtual Reality, MAVR
l 51 Markers acquired(cf. MPEG-4 specifications). l 7 virtual markers definedthrough the LRF (green). l 2 Virtual markers definedthrough Real Markers (blue). l 56 control points for the mesh + 4 for LRF. 26/49
http://homes.dsi.unimi .it/~borghese/
13
A possible implementation of mesh deformation Model constituted of a 3D mesh, inspired to the anatomy. Goal: duplicate facial appearence with few parameters.
Mesh warping is induced by the modification (of the position of) few features. The modification consists in the change in 3D position of the features. The modified mesh is then rendered.
Laboratory of Motion Analysis & Virtual Reality, MAVR
27/49
http://homes.dsi.unimi .it/~borghese/
Disgust
Laboratory of Motion Analysis & Virtual Reality, MAVR
28/49
http://homes.dsi.unimi .it/~borghese/
14
Fear
Laboratory of Motion Analysis & Virtual Reality, MAVR
29/49
http://homes.dsi.unimi .it/~borghese/
Anger
Laboratory of Motion Analysis & Virtual Reality, MAVR
30/49
http://homes.dsi.unimi .it/~borghese/
15
Surprise
Laboratory of Motion Analysis & Virtual Reality, MAVR
31/49
http://homes.dsi.unimi .it/~borghese/
Sadness
Laboratory of Motion Analysis & Virtual Reality, MAVR
32/49
http://homes.dsi.unimi .it/~borghese/
16
Happiness
Laboratory of Motion Analysis & Virtual Reality, MAVR
33/49
http://homes.dsi.unimi .it/~borghese/
Direct parameterization Universal model (e.g. Parke’s model, 1974) + few parameters to adapt the model and obtain “key poses” or “animation curves”.
The time course of the parameters can be given or derived from motion capture. Complexity of the face, from the kinematics / deformation point of view, is captured by the mesh (points + connectivity).
Laboratory of Motion Analysis & Virtual Reality, MAVR
34/49
http://homes.dsi.unimi .it/~borghese/
17
Expressive structure of the face •Emotion expression. Mainly in the eyes, eye-brows and mouth. •Somatic expressions: pain, sleepness, hungry, attention, shock…
Laboratory of Motion Analysis & Virtual Reality, MAVR
35/49
http://homes.dsi.unimi .it/~borghese/
Some of the faces of Paul Ekman
Laboratory of Motion Analysis & Virtual Reality, MAVR
36/49
http://homes.dsi.unimi .it/~borghese/
18
How FACS was developed •The main idea was to determine which muscles can be activated indipendently and determine how these muscles modify the appearance of the face. •Goal is to identify elementary motion associated to each elementary action (Action Unit): many muscles contribute to the single elementary action. •The corrispondence between muscles and Action Units is many to many. •The identified Action Units are 46. They are activated in different percentage in each expression è They are added to produce a given facial expression. •Problems are in the description of jaw and lips motion.
37/49
http://homes.dsi.unimi .it/~borghese/
Laboratory of Motion Analysis & Virtual Reality, MAVR
38/49
http://homes.dsi.unimi .it/~borghese/
The Action Units (AU)
Laboratory of Motion Analysis & Virtual Reality, MAVR
19
Avenues of research Detailed biomechanical models (FEM). Not compatible with real-time for non-linear elements. Streaming of images over the 3D mesh. Blending 3D models of “critical” parts (tongue, teeth..) and pre-defined texture for grooves (bump mapping) with the 3D mesh. Map feature or marker motion into FACS => Animate a “physical” mesh. Intersting problems: Impossible interviews. Virtual speakers for low-band transmission. Rehabilitation. Laboratory of Motion Analysis & Virtual Reality, MAVR ….. 39/49
http://homes.dsi.unimi .it/~borghese/
Outline Introduction: what is Motion Capture? History and Motion Capture technologies. Passive Markers Motion Capture. Specialized motion capture: hand, gaze and face. From Motion Capture to Animation (post-processing).
Laboratory of Motion Analysis & Virtual Reality, MAVR
40/49
http://homes.dsi.unimi .it/~borghese/
20
Avatars and Motion Capture Jacks
Avatars are goods from the heaven (from Induism, usually Visnù)
http://www.plmsolutions-eds.com/products/efactory/jack/moviesandimages.shtml
Laboratory of Motion Analysis & Virtual Reality, MAVR
41/49
http://homes.dsi.unimi .it/~borghese/
The human skeleton has complex articulations “Rigid” bones connected. Tendons keep the bones in place. Motion allowed can be very complex (e.g. shoulder, spine). The reconstruction of the finest details of the motion are beyond reach, simplifying assumptions are made => Level of detail in motion analysis
Laboratory of Motion Analysis & Virtual Reality, MAVR
42/49
http://homes.dsi.unimi .it/~borghese/
21
Retargetting From Motion Capture to Virtual Motion: 3D positions → Angles Model fitting Motion correction
Laboratory of Motion Analysis & Virtual Reality, MAVR
43/49
http://homes.dsi.unimi .it/~borghese/
Motion correction & retargetting What happens if the arm of the digital character enter inside the shoulder of his girl- friend? The problem is reframes as an optimal control problem. Zero error in the final frame. Minimal deviation of the control actions (the angle sequence).
a ∑k (u k (t )) + b( x d (t N ) − x(t N )) 2 2
{ u ( t k )}
Hard and Soft constraints Laboratory of Motion Analysis & Virtual Reality, MAVR
44/49
http://homes.dsi.unimi .it/~borghese/
22
Motion retargetting: an example
Data captured have to be adapted to a smaller female.
Laboratory of Motion Analysis & Virtual Reality, MAVR
45/49
http://homes.dsi.unimi .it/~borghese/
Motion retargetting: an example
Laboratory of Motion Analysis & Virtual Reality, MAVR
46/49
http://homes.dsi.unimi .it/~borghese/
23
Clinical Motion Analysis
MOTION ANALYSER
JOINT KINEMATICS
FORCE TRANSDUCER
JOINT KINETICS EXTERNAL FORCES
MATHEMATICAL MODELS
PLANTAR PRESSION
EMG
MUSCLE ACTIVATION AND FORCE
Laboratory of Motion Analysis & Virtual Reality, MAVR
47/49
http://homes.dsi.unimi .it/~borghese/
The future Digital and Reality in real- time (virtual theater). Color-coded markers. Mixed vision/marker techniques. Integration of gloves, gaze trackers and marker trackers. Detailed biomechanical models. More biology into digital characters (motion retargetting, with “biological rules”). Is there any future for motion capture? Laboratory of Motion Analysis & Virtual Reality, MAVR
48/49
http://homes.dsi.unimi .it/~borghese/
24
Motion capture
Laboratory of Motion Analysis & Virtual Reality, MAVR
49/49
http://homes.dsi.unimi .it/~borghese/
25
