Augmented and Mixed Reality Applications at VTT Charles Woodward April 16, 2010
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Mobile Augmented Reality one of the technologies to
“most likely to alter industries, fields of research, and the way we live”. - MIT’s annual review 10 Emerging Technologies 2007 - Gartner’s Top Ten Disruptive Technologies 2008-2012 - First consumer applications on camera phones 2009 - Commercial breakthrough predicted 2010
1
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
INTRODUCTION
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
VTT Technical Research Centre of Finland Largest contract research instute in Northern Europe Basic Producing research, research development, testing and information services Serving public sector and companies as well as international organizations
Figures
Applied research Development
IV UN
ES ITI S ER
VTT IN DU ST RY
work 2700 employees 245 M€ turnover Funding1/3 government, 1/3 industry, 1/3 jointly funded
2
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Virtual and Augmented Reality at VTT General VR/AR expertise around VTT 14 research teams in VR/AR roadmap work Applications from games to digital media, construction to production, ships, cars, military, … Technology incl. core VR/AR solutions, sensors, GSM/GPS positioning, multimodality, 3D audio, motion tracking, ubi & ambient intelligence, Special strengths: Multi-disciplinary research, strong industry relations and experience
Augmented Reality team Research personnel 9+ people AR core technology & applications
AR at VTT – situation today AR driven projects > 2 M€ annually + VTT coordinated 10 M€ EU project ManuVAR Overall, AR is now bigger than VR
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Introduction / AR Related technologies
Augmented Reality (AR) Mixes virtual objects with view of real world C.f. Virtual Reality (completely virtual worlds) Properties: real-time, interactive, 3D Tracking + rendering + display
Reality
Augmented Reality
Augmented Virtuality Mediated Reality Diminished Reality -> Mixed Reality (MR)
Augmented Virtuality
Virtual Reality
Mixed Reality
3
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Example Application ARCatalogue Viewing with data glasses / web camera Stereo 3D products appear from catalogue Based on marker detection from camera view
Applications, e.g. B2B marketing (by Lappset Group ) 3D viewing of engineering drawings (by Pöyry Inc.)
Also, camera phones Consumer solutions
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Dibidogs 3D animated chirdren’s TV series By Futurecode (Finland) and Blue Arc (China) Launched April 2nd, 2010, MTV3 / SubTV Junior
Augmented magazine advertisement TV-magazines Katso and TVSeiska, March 2010 3D animations appear augmented from magazine Interaction with augmented Dibidog
4
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Dibidogs, Next Steps Mobile phone implementation Introduced at Sanghai World Expo, May 1st, 2010 Nokia Maemo (N900) and Symbian S60 platforms
Augmented childrens’ storybook Kustannus Paasilinna, August 2010 Various new interactions and narratives Markerless tracking, photorealistic rendering etc.
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
ALVAR Subroutine Library ”A Library for Virtual and Augmented Reality” ”AR engine” developed by VTT Free trial download at www.vtt.fi/multimedia/alvar.html Users e.g. Columbia Univ. (Goblin XNA), MIT (6th Sense)
Detecting and tracking 2D markers Accurate marker pose estimation, with erro concealing “Marker fields” can be defined manually, or autocalibrated Hiding of markers as sepcial feature
Utilities Camera calibration; Correcting lense undistortions; Optical flow tracking; Several basic filters and Kalman library; Finding exterior orientation of point-sets
Coming up Markerless tracking; Rendering module; Mobile phone implementation
5
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
ENTERTAINMENT AND GAMES
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Entertainment Application - CamBall Augmented networked table tennis Played with real rackets, color marker Video image of opponent shown at other end of table Virtual ball, table and net
Features Ball speed, racket size can be adjusted Solo game against a wall, or against "oneself" Game viewing by Internet audience Recording of game
Benefits Socially and physically activating Natural human-computer user interaction Fits in small interiors, at home or office Suits also for kids and elderly people No special hardware required
6
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Mobile Entertainment - SymBall Virtual table tennis game for camera phones Camera driven game control by feature/motion detection Data connection between players via Bluetooth or GPRS First networked camera driven game in the world
Implementation Symbian OS/Series 60 (& J2ME) Features: sounds, ball speed, etc. 3D stereo game using r/b goggles!
Future plan: Augmented version Using turnable camera, face detection Show real opponent on phone display
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Experimental Work ”TelePet” Augmented Networked Games for Dogs To entertain dogs over Internet during master’s absence Game control based on gesture detection
Example game Master throws virtual ball, dog has to “catch” Rewards, e.g. biscuit box operated over internet Initial animal tests - positive results
Internet
7
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Vappunaamari.com Virtual masks for May Day party Face detection/augmenting on camera phones Provided as public service, > 1500 downloads Features: send photo to friends, user created masks, compressing augmented video, …
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
INTERIOR DESIGN
8
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Augmented Interior Design – Alternative Solutions Basic solution Virtual furniture object tied to each marker Interior designed by moving markers on the floor Display on PC screen or data glasses
Benefits Furniture is ”light” to move around Easy to try out different alternatives No modelling of real environment required
Things to improve … Web camera … isn’t very mobile Data glasses / PC .. not too convenient
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Augmenting in Digital Camera Images Application ”ARInteriors” Single marker placed on floor, defining scale and perspective Digital camera images of the room are uploaded to computer Virtual furniture is selected and placed with menu/mouse Features: lighting, shadows, hiding of marker etc.
Benefits Hardware (camera, PC) found in every home Better image quality, resolution, accuracy etc. Photorealistic 3D rendering
9
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Augmenting on TabletPC Application “ARMobile” Tablet PC (Sony Vaio), camera integrated Optional data glasses User interface to fit small screen, stylus input No configuration files required Arbitrary marker orientations Marker fields calculated automatically Moving, scaling and rotating objects Adjusting of lights, with soft shadows Erasing of markers Storing of video clip & project Kalman filter & other stabilisation Feature based tracking when markers disappear
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Augmenting on Mobile Phones Application ”ARPhone” Placing virtual furniture on phone’s camera image Models are loaded using matrix code in product catalogue or advert User places the furniture in camera view by 3D rotation and zoom Furniture stays in place when moving the camera phone Based on feature tracking - no markers required Also, augmenting of real furniture
Find appealing furniture in ad
Download virtual furniture via matrix code
Instantly view furniture at home on mobile phone
Verify by high-end PC solution (if required)
Get to shop, purchase product
10
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
ARStudio Unified solution for general AR applications Formats: real time / stored video still images Single/multimarker tracking -> markerless methods Matching augmented content with video quality Photorealism, shadows, materials, lighting, …
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
VividAR AR interior design solution by VividWorks Based on VTT’s technology Launched February 2010 Consumer web solution later this spring
11
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
ARCHITECTURE AND CONSTRUCTION
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Augmented Reality in Architecture Case Study: VTT’s Digitalo offices Digitalo = “DigiBuilding” Construction started Dec 2003 Building ready Oct 2005
Various AR/VR methods used Radiosity visualization 3D Landscapes CAVE Virtual Reality Augmented Scale Model Augmented Web Camera Mobile Augmented Reality
Projects involved Commissions by Senate Properties Project ARARAT 2004-2006
12
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Augmented Scale Model Digitalo augmented on conference table Viewing through video glasses (stereo) Move around model, communicate with people
Features Virtual menu for user interaction Virtual reality tour utilising feature based tracking Predefined walk-through routes, with hyperlinks Realism: textures, reflections, shadows etc. Add, move and remove parts (AR-catalogue)
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Augmented Web Camera Digitalo augmented in live web camera view Presentation for future users of building Follow-up and comparison with original plans View with or without virtual building Real-time interaction (zoom, pan, etc.) Viewing inside to the building, etc.
Ongoing development (project AR4BC) Integrating 4D IFC models to live web cameras
13
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Mobile Augmented Reality Outdoors visualization of Digitalo Pioneering work by VTT 2003
Handheld display device PDA + WLAN + camera Lightweight, “user friendly”, inexpensive Allows frozen view, multiple users, etc.
Client/server solution Server laptop PC for 3D computation Streaming video between blient and server Speed 2 frames / sec
Outdoors tracking Using large 80 x 80 cm markers Mobility by moving markers around
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Unprepared Environments Markerless implementation “AROnSite” Feature based tracking Interactive positioning of building to real scene Visualization in any unprepared environment
Applications, e.g. Helsinki Music Hall visualization 2006
14
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
“Google Earth on Earth” Google Earth + AROnSite Building placed in Google Earth Semi-automatic positioning using GPS
Current applications, e.g. Paper mill visualizations by Pöyry Excavator visualization by Metso Skanska new Helsinkk offices
Future work Automatic positioning (direction) Photorealism: e.g. lighting, shadows, masking Etc., being implemented in project AR4BC (see next slide)
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Project “AR4BC” Augmented Reality for Building and Construction vertical representation of Finnish & international B&C sector duration 2008 – 2010 budget 420 k€ Physical model authors • • • •
Structural Mechanical Architectural Electrical
Process model authors • • • •
Schedule Classification Logistics Safety
Information users • • • • • •
View Review Comments Approvals Status Feedback
Building Information Models (BIM) Reality
Augmented Reality
Augmented Virtuality
Virtual Reality
Mixed Reality
15
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Goals Independent of Google Earth data formats (Collada) Support for general/proprietary maps (cf. inaccuracy of GE maps) Support for Building Information Models (e.g. IFC) Support for time component, 4D (e.g. MS Project) Support for several building models at a time Photorealistic visualization (CG tuned for AR) User interface (wxWidgets & new features) Visual feedback to design system
Augmented Reality + Augmented Virtuality = Mixed Reality
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
INDUSTRIAL APPLICATIONS
16
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Power Plants Remote Maintenance Tasks ITER fusion reactor mock-up at Tampere Maintenance robot augmented / animated inside test platform Application e.g. for discrepancy checking (after accident or other changes)
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Project ”NOSE” “Nomadic Use of a Plant Model” 2007-2009, budget > 700 k€ Motto: “Learning by Walking”
Partners: Pöyry, Metso, Wärtsilä, Fortum, Intergraph, Nokia, …
17
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Simantics Mobile Studio
NOSE Implementation Simantics Mobile Mobile AR user interface Studio for content creation Mobile database & Internet
Simantics Mobile
Locationing technology Pseudosatellites (GPS inside factory hall), WLAN, gait etc. Model & template based tracking Sensor fusion
Pöyry Virtual Mill
Applications, e.g.
VTT Knowpap
Mobile training Mobile operation Mobile maintenance Simantics Mobile Database
Metso DNA OPC Server
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Project ”AugAsse” “Increasing Efficiency in Assembly Work by Augmented Reality” Duration 2006-2008, budget 800+ k€ Partners Valtra, Metso, Wärtstilä. Kalmar. Sandvik, …
Implementation phases 3D Puzzle, multimodal interaction Camera phone implementation, client/server Industrial case: hydraulic block by Valtra Plc Incl. data pipeline CAD -> AR, user studies etc.
18
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Data Pipeline Data pipeline from 3D CAD to augmented assembly CATIA -> STEP -> Reduced B-rep -> Design for Assembly -> Animations -> Augmenting
Work Phase:23 Return tube Tool: 27 mm wrench
s ter s me u re ara fig n p ilit y ctio bla b u m od Pr asse d an
ly mb se As
e tur uc st r
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Mobile Phone Implementation Client/server easy to implement, general solution tracking and other heavy computation can be done on server no need to store large model data on mobile device using subimages, no need for any 3D support on mobile device
Still images in most cases, the assembly station is static it is sufficient that the augmented content is “live” it is more ergonomic to view the image without having to point at target robust, provides good image size (accuracy) for tracking enables general implementation, e.g. on Java ME
19
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
AugAsse User Study Group Statistics
Assembling duration (h:mm:ss) Observed frequency of using a wrong tool Observed frequency of false positioning
Instruction type paper instructions augmented-reality instructions paper instructions augmented-reality instructions paper instructions augmented-reality instructions
30
Mean 0:36:34
Std. Deviation 0:07:42
Std. Error Mean 0:01:24
29
0:31:45
0:04:31
0:00:50
26
1,96
1,455
,285
25
,32
,627
,125
26
3,54
1,964
,385
25
1,48
1,873
,375
N
Assembling is on average quicker with animated instructions and results in less mistakes than with paper instructions. The usage of wrong tool and false positioning are on average less frequent with augmented-reality instructions than with paper ones. A target for improvement is the HMD hardware. Operation with the helmet and wires was often experienced uncomfortable.
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Project ”ManuVAR” ”Manual Work Support Throughout System Lifecycle by VR/AR” EU FP7 NMP “Beyond lean manufacturing” 18 partners, 70+ participants, VTT as coordinator Duration 2009-2012, budget 9.7M€
The general objective is to develop • (1) technology platform and • (2) methodological framework for supporting all aspects of manual work throughout the system lifecycle with the ultimate goal to 1. increase productivity and quality and reduce cost of manual work in the whole lifecycle 2. facilitate adaptation to product customization and changes 3. support efficient knowledge and skill management through the lifecycle
20
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
ManuVAR Applications
Thales Alenia Space
Satellite assembly
NEM Solutions
Remote training
TNO
Manufacturing design
Industrial plant maintenance Tecnatom
Maintenance of heavy machinery Metso Minerals
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
RECENT PROJECTS: MIXED REALITY
21
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Mobile Mixed Reality Co-creation project “MMR” with Nokia
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Mixed Reality Video Conference Teleconference btw real people and SL avatars Avatars appear augmented in real room People’s gestures are replicated by avatars Collaboration with virtual objects Partners VTT, TKK, Nokia, IBM
22
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Multitouch Interaction Multitouch table developed by VTT LCD display with cameras in the corners Any flat surface can be used for multitouch interaction Application example: connecting floor plan with CAVE, navigation in 3D architectural model At Sanghai World Expo 2010: multitouch in the air, to control giant projector walls, tens of people Collaborative work, cf. “Minority Report”
VTT TECHNICAL RESEARCH CENTRE OF FINLAND
Thank You! Dr. Charles Woodward VTT Technical Research Centre of Finland Vuorimiehentie 3, Espoo, Finland Tel: +358 40 500 1514
[email protected] http://www.vtt.fi/multimedia
23