Augmented and Mixed Reality Applications at VTT Charles Woodward April 16, 2010

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Mobile Augmented Reality one of the technologies to

“most likely to alter industries, fields of research, and the way we live”. - MIT’s annual review 10 Emerging Technologies 2007 - Gartner’s Top Ten Disruptive Technologies 2008-2012 - First consumer applications on camera phones 2009 - Commercial breakthrough predicted 2010

1

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

INTRODUCTION

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

VTT Technical Research Centre of Finland Largest contract research instute in Northern Europe Basic Producing research, research development, testing and information services Serving public sector and companies as well as international organizations

Figures

Applied research Development

IV UN

ES ITI S ER

VTT IN DU ST RY

work 2700 employees 245 M€ turnover Funding1/3 government, 1/3 industry, 1/3 jointly funded

2

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Virtual and Augmented Reality at VTT General VR/AR expertise around VTT 14 research teams in VR/AR roadmap work Applications from games to digital media, construction to production, ships, cars, military, … Technology incl. core VR/AR solutions, sensors, GSM/GPS positioning, multimodality, 3D audio, motion tracking, ubi & ambient intelligence, Special strengths: Multi-disciplinary research, strong industry relations and experience

Augmented Reality team Research personnel 9+ people AR core technology & applications

AR at VTT – situation today AR driven projects > 2 M€ annually + VTT coordinated 10 M€ EU project ManuVAR Overall, AR is now bigger than VR

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Introduction / AR Related technologies

Augmented Reality (AR) Mixes virtual objects with view of real world C.f. Virtual Reality (completely virtual worlds) Properties: real-time, interactive, 3D Tracking + rendering + display

Reality

Augmented Reality

Augmented Virtuality Mediated Reality Diminished Reality -> Mixed Reality (MR)

Augmented Virtuality

Virtual Reality

Mixed Reality

3

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Example Application ARCatalogue Viewing with data glasses / web camera Stereo 3D products appear from catalogue Based on marker detection from camera view

Applications, e.g. B2B marketing (by Lappset Group ) 3D viewing of engineering drawings (by Pöyry Inc.)

Also, camera phones Consumer solutions

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Dibidogs 3D animated chirdren’s TV series By Futurecode (Finland) and Blue Arc (China) Launched April 2nd, 2010, MTV3 / SubTV Junior

Augmented magazine advertisement TV-magazines Katso and TVSeiska, March 2010 3D animations appear augmented from magazine Interaction with augmented Dibidog

4

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Dibidogs, Next Steps Mobile phone implementation Introduced at Sanghai World Expo, May 1st, 2010 Nokia Maemo (N900) and Symbian S60 platforms

Augmented childrens’ storybook Kustannus Paasilinna, August 2010 Various new interactions and narratives Markerless tracking, photorealistic rendering etc.

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

ALVAR Subroutine Library ”A Library for Virtual and Augmented Reality” ”AR engine” developed by VTT Free trial download at www.vtt.fi/multimedia/alvar.html Users e.g. Columbia Univ. (Goblin XNA), MIT (6th Sense)

Detecting and tracking 2D markers Accurate marker pose estimation, with erro concealing “Marker fields” can be defined manually, or autocalibrated Hiding of markers as sepcial feature

Utilities Camera calibration; Correcting lense undistortions; Optical flow tracking; Several basic filters and Kalman library; Finding exterior orientation of point-sets

Coming up Markerless tracking; Rendering module; Mobile phone implementation

5

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

ENTERTAINMENT AND GAMES

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Entertainment Application - CamBall Augmented networked table tennis Played with real rackets, color marker Video image of opponent shown at other end of table Virtual ball, table and net

Features Ball speed, racket size can be adjusted Solo game against a wall, or against "oneself" Game viewing by Internet audience Recording of game

Benefits Socially and physically activating Natural human-computer user interaction Fits in small interiors, at home or office Suits also for kids and elderly people No special hardware required

6

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Mobile Entertainment - SymBall Virtual table tennis game for camera phones Camera driven game control by feature/motion detection Data connection between players via Bluetooth or GPRS First networked camera driven game in the world

Implementation Symbian OS/Series 60 (& J2ME) Features: sounds, ball speed, etc. 3D stereo game using r/b goggles!

Future plan: Augmented version Using turnable camera, face detection Show real opponent on phone display

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Experimental Work ”TelePet” Augmented Networked Games for Dogs To entertain dogs over Internet during master’s absence Game control based on gesture detection

Example game Master throws virtual ball, dog has to “catch” Rewards, e.g. biscuit box operated over internet Initial animal tests - positive results

Internet

7

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Vappunaamari.com Virtual masks for May Day party Face detection/augmenting on camera phones Provided as public service, > 1500 downloads Features: send photo to friends, user created masks, compressing augmented video, …

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

INTERIOR DESIGN

8

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmented Interior Design – Alternative Solutions Basic solution Virtual furniture object tied to each marker Interior designed by moving markers on the floor Display on PC screen or data glasses

Benefits Furniture is ”light” to move around Easy to try out different alternatives No modelling of real environment required

Things to improve … Web camera … isn’t very mobile Data glasses / PC .. not too convenient

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmenting in Digital Camera Images Application ”ARInteriors” Single marker placed on floor, defining scale and perspective Digital camera images of the room are uploaded to computer Virtual furniture is selected and placed with menu/mouse Features: lighting, shadows, hiding of marker etc.

Benefits Hardware (camera, PC) found in every home Better image quality, resolution, accuracy etc. Photorealistic 3D rendering

9

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmenting on TabletPC Application “ARMobile” Tablet PC (Sony Vaio), camera integrated Optional data glasses User interface to fit small screen, stylus input No configuration files required Arbitrary marker orientations Marker fields calculated automatically Moving, scaling and rotating objects Adjusting of lights, with soft shadows Erasing of markers Storing of video clip & project Kalman filter & other stabilisation Feature based tracking when markers disappear

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmenting on Mobile Phones Application ”ARPhone” Placing virtual furniture on phone’s camera image Models are loaded using matrix code in product catalogue or advert User places the furniture in camera view by 3D rotation and zoom Furniture stays in place when moving the camera phone Based on feature tracking - no markers required Also, augmenting of real furniture

Find appealing furniture in ad

Download virtual furniture via matrix code

Instantly view furniture at home on mobile phone

Verify by high-end PC solution (if required)

Get to shop, purchase product

10

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

ARStudio Unified solution for general AR applications Formats: real time / stored video still images Single/multimarker tracking -> markerless methods Matching augmented content with video quality Photorealism, shadows, materials, lighting, …

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

VividAR AR interior design solution by VividWorks Based on VTT’s technology Launched February 2010 Consumer web solution later this spring

11

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

ARCHITECTURE AND CONSTRUCTION

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmented Reality in Architecture Case Study: VTT’s Digitalo offices Digitalo = “DigiBuilding” Construction started Dec 2003 Building ready Oct 2005

Various AR/VR methods used Radiosity visualization 3D Landscapes CAVE Virtual Reality Augmented Scale Model Augmented Web Camera Mobile Augmented Reality

Projects involved Commissions by Senate Properties Project ARARAT 2004-2006

12

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmented Scale Model Digitalo augmented on conference table Viewing through video glasses (stereo) Move around model, communicate with people

Features Virtual menu for user interaction Virtual reality tour utilising feature based tracking Predefined walk-through routes, with hyperlinks Realism: textures, reflections, shadows etc. Add, move and remove parts (AR-catalogue)

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Augmented Web Camera Digitalo augmented in live web camera view Presentation for future users of building Follow-up and comparison with original plans View with or without virtual building Real-time interaction (zoom, pan, etc.) Viewing inside to the building, etc.

Ongoing development (project AR4BC) Integrating 4D IFC models to live web cameras

13

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Mobile Augmented Reality Outdoors visualization of Digitalo Pioneering work by VTT 2003

Handheld display device PDA + WLAN + camera Lightweight, “user friendly”, inexpensive Allows frozen view, multiple users, etc.

Client/server solution Server laptop PC for 3D computation Streaming video between blient and server Speed 2 frames / sec

Outdoors tracking Using large 80 x 80 cm markers Mobility by moving markers around

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Unprepared Environments Markerless implementation “AROnSite” Feature based tracking Interactive positioning of building to real scene Visualization in any unprepared environment

Applications, e.g. Helsinki Music Hall visualization 2006

14

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

“Google Earth on Earth” Google Earth + AROnSite Building placed in Google Earth Semi-automatic positioning using GPS

Current applications, e.g. Paper mill visualizations by Pöyry Excavator visualization by Metso Skanska new Helsinkk offices

Future work Automatic positioning (direction) Photorealism: e.g. lighting, shadows, masking Etc., being implemented in project AR4BC (see next slide)

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Project “AR4BC” Augmented Reality for Building and Construction vertical representation of Finnish & international B&C sector duration 2008 – 2010 budget 420 k€ Physical model authors • • • •

Structural Mechanical Architectural Electrical

Process model authors • • • •

Schedule Classification Logistics Safety

Information users • • • • • •

View Review Comments Approvals Status Feedback

Building Information Models (BIM) Reality

Augmented Reality

Augmented Virtuality

Virtual Reality

Mixed Reality

15

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Goals Independent of Google Earth data formats (Collada) Support for general/proprietary maps (cf. inaccuracy of GE maps) Support for Building Information Models (e.g. IFC) Support for time component, 4D (e.g. MS Project) Support for several building models at a time Photorealistic visualization (CG tuned for AR) User interface (wxWidgets & new features) Visual feedback to design system

Augmented Reality + Augmented Virtuality = Mixed Reality

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

INDUSTRIAL APPLICATIONS

16

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Power Plants Remote Maintenance Tasks ITER fusion reactor mock-up at Tampere Maintenance robot augmented / animated inside test platform Application e.g. for discrepancy checking (after accident or other changes)

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Project ”NOSE” “Nomadic Use of a Plant Model” 2007-2009, budget > 700 k€ Motto: “Learning by Walking”

Partners: Pöyry, Metso, Wärtsilä, Fortum, Intergraph, Nokia, …

17

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Simantics Mobile Studio

NOSE Implementation Simantics Mobile Mobile AR user interface Studio for content creation Mobile database & Internet

Simantics Mobile

Locationing technology Pseudosatellites (GPS inside factory hall), WLAN, gait etc. Model & template based tracking Sensor fusion

Pöyry Virtual Mill

Applications, e.g.

VTT Knowpap

Mobile training Mobile operation Mobile maintenance Simantics Mobile Database

Metso DNA OPC Server

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Project ”AugAsse” “Increasing Efficiency in Assembly Work by Augmented Reality” Duration 2006-2008, budget 800+ k€ Partners Valtra, Metso, Wärtstilä. Kalmar. Sandvik, …

Implementation phases 3D Puzzle, multimodal interaction Camera phone implementation, client/server Industrial case: hydraulic block by Valtra Plc Incl. data pipeline CAD -> AR, user studies etc.

18

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Data Pipeline Data pipeline from 3D CAD to augmented assembly CATIA -> STEP -> Reduced B-rep -> Design for Assembly -> Animations -> Augmenting

Work Phase:23 Return tube Tool: 27 mm wrench

s ter s me u re ara fig n p ilit y ctio bla b u m od Pr asse d an

ly mb se As

e tur uc st r

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Mobile Phone Implementation Client/server easy to implement, general solution tracking and other heavy computation can be done on server no need to store large model data on mobile device using subimages, no need for any 3D support on mobile device

Still images in most cases, the assembly station is static it is sufficient that the augmented content is “live” it is more ergonomic to view the image without having to point at target robust, provides good image size (accuracy) for tracking enables general implementation, e.g. on Java ME

19

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

AugAsse User Study Group Statistics

Assembling duration (h:mm:ss) Observed frequency of using a wrong tool Observed frequency of false positioning

Instruction type paper instructions augmented-reality instructions paper instructions augmented-reality instructions paper instructions augmented-reality instructions

30

Mean 0:36:34

Std. Deviation 0:07:42

Std. Error Mean 0:01:24

29

0:31:45

0:04:31

0:00:50

26

1,96

1,455

,285

25

,32

,627

,125

26

3,54

1,964

,385

25

1,48

1,873

,375

N

Assembling is on average quicker with animated instructions and results in less mistakes than with paper instructions. The usage of wrong tool and false positioning are on average less frequent with augmented-reality instructions than with paper ones. A target for improvement is the HMD hardware. Operation with the helmet and wires was often experienced uncomfortable.

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Project ”ManuVAR” ”Manual Work Support Throughout System Lifecycle by VR/AR” EU FP7 NMP “Beyond lean manufacturing” 18 partners, 70+ participants, VTT as coordinator Duration 2009-2012, budget 9.7M€

The general objective is to develop • (1) technology platform and • (2) methodological framework for supporting all aspects of manual work throughout the system lifecycle with the ultimate goal to 1. increase productivity and quality and reduce cost of manual work in the whole lifecycle 2. facilitate adaptation to product customization and changes 3. support efficient knowledge and skill management through the lifecycle

20

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

ManuVAR Applications

Thales Alenia Space

Satellite assembly

NEM Solutions

Remote training

TNO

Manufacturing design

Industrial plant maintenance Tecnatom

Maintenance of heavy machinery Metso Minerals

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

RECENT PROJECTS: MIXED REALITY

21

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Mobile Mixed Reality Co-creation project “MMR” with Nokia

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Mixed Reality Video Conference Teleconference btw real people and SL avatars Avatars appear augmented in real room People’s gestures are replicated by avatars Collaboration with virtual objects Partners VTT, TKK, Nokia, IBM

22

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Multitouch Interaction Multitouch table developed by VTT LCD display with cameras in the corners Any flat surface can be used for multitouch interaction Application example: connecting floor plan with CAVE, navigation in 3D architectural model At Sanghai World Expo 2010: multitouch in the air, to control giant projector walls, tens of people Collaborative work, cf. “Minority Report”

VTT TECHNICAL RESEARCH CENTRE OF FINLAND

Thank You! Dr. Charles Woodward VTT Technical Research Centre of Finland Vuorimiehentie 3, Espoo, Finland Tel: +358 40 500 1514 [email protected] http://www.vtt.fi/multimedia

23