Advanced Personal Robot Interaction Research Dr. Jiqiang Song Intel Labs China [email protected]

1

Heading from CT to RT era

RT = CT + Robot Technology

Computer Technology

Sensing

Perception

AI

Cognition

ME

Action

user

Artificial Intelligence

Mechanical Electronics

2

Human-robot Interaction (HRI)

What is an ideal personal robot? • Siri*？ • NAO*？ • Atlas*？ • Pepper*？ • Prof. Ishiguro’s robot clone? • DLR* Rollin’ Justin*？ Image source:

3

https://www.aldebaran.com/en

http://www.eng.osaka-u.ac.jp/en/index.html

http://www.bostondynamics.com/

http://www.dlr.de/rmc/rm/en/desktopdefault.aspx/tabid-5471/

We count on robots to solve BIG social problems China demographics at the year 2010

HSR

As of today, 212M Age > 60 Keep increasing 15 years to the peak: 350M

Image source: https://en.wikipedia.org/wiki/ Demographics_of_China 4

Three essentials for personal robot booming Useful

Affordable

￥ Specific functions and usages

5

Reliable

$

Reasonable price & business model

Safe, durable & privacy preserving

Three stages of personal robots We are here!

Connected • • • •

6

Wireless com Home gateway Information delivery Social service

Smart • • • •

Listen & speak See & recognize Telepresence Personalization

Autonomous • • • •

Behavior understanding Emotion understanding Reasoning & planning Reliable & predictable

From Smart to Autonomous: four key pieces Sensor fusion

Smart computing acceleration

7

Distributed heterogeneous computing

Safe & Reliable

#1: Sense beyond human

8

Sensor fusion

Distributed heterogeneous computing

Smart computing acceleration

Safe & Reliable

Intel® RealSenseTM Product (2016)

9

3D Sensing with Intel® RealSenseTM Technology

10

Limitation of Single Sensor

11

Out of range Reflective Dark Material

Image source: https://homes.cs.washington.edu/~xren/publication/3d-mapping-iser-10-final.pdf

Need more sensor input to improve navigation Robot Navigation LiDAR

12

U/S range sensor

IR range sensor

IMU

Odometer

SLAM Fusion for Robot Navigation • Deliver flexible & affordable autonomous navigation solutions in complex home environment

LiDAR UWB VSLAM IMU/ Odom

Sensing 13

SLAM Fusion (Optimized Algorithm)

Localization and Mapping

ID/MD/HW integration 19

Planning and Exploration

Need more sensor input to improve HRI Robot Navigation LiDAR

U/S range sensor

IR range sensor

IMU

Odometer

Robot Interaction RGB camera

14

Depth camera

Microphone array

mmWave scanner

nSense fusion for human/object detection

RGB camera

15

Depth camera

mm-Wave scanner (30GHz)

#2: Compute in the right way

16

Sensor fusion

Distributed heterogeneous computing

Smart computing acceleration

Safe & Reliable

Distributed heterogeneous computing Low-power AOAC system (Always On Always Connected）

High performance smart computing system （CPU+GPU+FPGA）

Real-time motion control system (MCU) 17

Voice and wireless wakeup

AOAC

Voice command

18

Anomaly detection

High-performance smart computing system SIP

Configurable I/O

QPI/UPI

CPU

PCIE Prog I/F

FPGA AFU

Peripherals

PCIE

GPU

19

QPI: Quick Path Interconnect UPI: Ultra Path Interconnect Prog I/F: FPGA programming interface AFU: Accelerator Function Unit

FPGA acceleration for smart computing Applications

• FPGA - Sensing and acting

CNN SIFT

- Cognitive computation

• CPU - Planning and decision - Communication and service

DNN DTW

FFT

Face/Emotion Motion control SSD DDR

FPGA Fuse/Accelerate

Planning

Semantic scene

DDR

CPU PCIE

App/Service

Sensors

Actions 20

ORB

SURF

De-noise

LiDAR mm-Wave IMU MIC Array RealSense camera

Visual-SLAM

Emergency Hands Arms Wheels

20

Wi-Fi BT 5G

#3: Add more intelligence, and FAST! Sensor fusion

Smart computing acceleration

21

Distributed heterogeneous computing

Safe & Reliable

Computer Vision – What is it? Computer Vision (CV) is a field that includes methods for acquiring, processing, analyzing and understanding images and, in general, high-dimensional data from the real world in order to produce numerical or symbolic information Wikipedia.org

Imaging Acquire

Process

Pixel processing

Visual Understanding Analyze

Understand

Object processing

Sample Capabilities

22

Face Detection & Recognition

Emotion Recognition

Text Recognition in the wild

Object Classification/Recognition

Activity Classification/Recognition

Scene Classification/Understanding

Video Classification/Summarization



Classification: Person, Camera Detection Person Camera 22

Action: Taking pictures

Target = human: advanced facial technology

GenX Acceleration

Face detection, morphing

Helen

Happy Female Adult

Emotion recognition

SDK

Intel Leading Face Analysis Technology

23

Dynamic facial expression recognition Recognize 7 basic facial expressions: happy, surprise, fear, disgust, anger, sadness and neutral in real time (100 fps on Intel® CORE CPU)

24

Emotion Recognition in the Wild Challenge Intel won EmotiW2015 in the audio-video based task, and competitors included 74 teams (CMU, UIUC, MSR, etc.) across the world • Task 1: EmotiW 2015 AFEW dataset (train/validation/test) 723/383/539 movie clips with audio, 7 basic facial expressions completely shown in the wild • Task 2: EmotiW 2015 SFEW dataset (train/validation/test) 958/436/372 static images, 7 basic facial expressions completely shown in the wild

Overall Recognition Rate (%) on EmotiW2015 Test Set

25

Methods

AFEW

SFEW

Baseline

39.33

39.13

Winner 2014

50.37

N/A

Intel audiovisual/visual solution

53.80

55.38

Active Vision 3D Slam and Navigation • Offline mapping, Robust 2D/3D combined algorithm • Real-time navigation, active vision

Fast Profile register

Robot Motion

Facing people • Novel robot capability Robot Motion

Social & Environmental Perception

Robust

People detection

Potential target

Confirmation

26

Finding people • Robust 360 degree pose human detection (on sofa, close to wall) • Facing people + Face/Person features

Robot Motion

Following people • Long-term tracking • Robust - deal with distraction from other people

Target = object: from 2D to 3D visual recognition

RGB + Depth

3D Point Cloud (3D-PC)

Depth and 3D information can suppress illumination, occlusion & clutter difficulties in 2D… 27

3D Object Detection Unified framework for 3D Pose Estimation and Object Detection using deep learning • Expect near real-time processing speed on Intel® CORE CPU.

RGB-D pair

Region Proposal

Instance Segmentation

Proposal & Mask

Normal Extraction

Pose CNN

Image Cropping

Object Class & Matched Coarse Pose

Model Model3D-PC 3D-PCin inwhite green Fine Pose & Object Class

Proposal & Mask as Input

Iterative Closest Point (ICP)

Cropped 3-channel normal

Coarse Pose & Model as Initialization

Captured/ICP aligned 3D-PC in white/red Captured/ICP aligned 3D-PC in green/red Bed room picture source: NYU Depth Dataset V2, Indoor Segmentation and Support Inference from RGBD Images, ECCV 2012 28

Robot’s memory: go beyond deep-learning Environment Video, image

Conversation

29

Activity Health

Small data and no labels – hard for DL Day

Week

*,*,*,

*,*,

*,*,*,

Medication

**

*

**

Eating

*

**

*

*

*,*,

*,

*,*,*,

*,*,

Location

Drinking

Month

Year

*,

Health data

*,

*,

Audio event

*,

*,

Environment

**

*

**

*

Conversation

…..

..

…

.

…

Time

Goal: Discover patterns in behavior and analyze association, for behavior prediction, anomaly detection 30

AI + HRI will break through

Human-robot Interaction

Perception

CT + AI

Cognition Low confidence?

Action

31

user

Ambiguity? Ask right questions

#4: No one is hurt

32

Sensor fusion

Distributed heterogeneous computing

Smart computing acceleration

Safe & Reliable

Dangerous: mobile robot without security

33

Safe and reliable

Container SE Linux

Data security

Intel®SGX

Physical security

Intel®TXT

34

Privacy leak

Denial of service

Full-stack security mechanism Container 1

Container 2 Secret

Secret

App: ROS

Apps

Apps Safety Framework

OS: Linux

Container N

Safety Framework

System libs Linux kernel

Secret

…

Apps Safety Framework

SE Linux

Preserve privacy

Protect apps’ codes and data Access control

MAC Policy

System integrity HW: CPU + Chipsets

35

Intel® CPU

Intel®TXT

Intel®SGX

Integrated four pieces into open research platform

Samples & Tools Middleware System Software Hardware Open Research Platform

36

Our personal robot testbeds

SLAM &Navigation navigation 37

FullFull capability interaction

Armless HRI manipulation free

37

Zi-Long: 3D Robot Head for HRI experiments

HW: Core i5 NUC + RealSense R200 + MIC array + Pico projector

38

38

Summary: full-stack HRI research platform Security enhanced Linux/ROS Personal assistant

Kids education

Kids gaming

Tele-medical

Follow me

Patrol

Spatial AR

Home appliance automation

Telepresence

Elderly care

Entertainment

Surveillance

WebRTC

Node-RED SDK

Apps

SDK

Robot SDK Human activity understanding

Emotional intelligence

Scene recognition

People Following/finding/facing

SLAM

Exploration

Environment sensing

ASR

Body Recog.

Object Recog.

V-SLAM

Localization

Navigation

Robot manipulation

TTS

Face Recog..

Text Recog.

Gesture Recog.

Trusted execution Environment Mobile base Wheels Lidar 39

Personalization

Middleware

Virtual Sensor

Capabilities

Algorithms

Middleware

Head

Body UWB

Arms

BLE/WiFi

Camera

RealSense

IMU

Other sensors

Projector /LED

MIC ARRAY

Project or/LED

Trusted Execution Technology

FPGA Acceleration

HW

What will you develop? 40

Legal Notices and Disclaimers Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation.Learn more at intel.com, or from the OEM or retailer. No computer system can be absolutely secure. Tests document performance of components on a particular test, in specific systems.Differences in hardware, software, or configuration will affect actual performance.Consult other sources of information to evaluate performance as you consider your purchase.For more complete information about performance and benchmark results, visit http://www.intel.com/performance. Cost reduction scenarios described are intended as examples of how a given Intel-based product, in the specified circumstances and configurations, may affect future costs and provide cost savings.Circumstances will vary.Intel does not guarantee any costs or cost reduction. This document contains information on products, services and/or processes in development.All information provided here is subject to change without notice.Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps. Statements in this document that refer to Intel’s plans and expectations for the quarter, the year, and the future, are forward-looking statements that involve a number of risks and uncertainties.A detailed discussion of the factors that could affect Intel’s results and plans is included in Intel’s SEC filings, including the annual report on Form 10-K. The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications.Current characterized errata are available on request. No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document. Intel does not control or audit third-party benchmark data or the web sites referenced in this document.You should visit the referenced web site and confirm whether referenced data are accurate. Intel, Xeon, Core, RealSense, and the Intel logo are trademarks of Intel Corporation in the United States and other countries.

*Other names and brands may be claimed as the property of others. © 2016 Intel Corporation.

41

Risk Factors The above statements and any others in this document that refer to future plans and expectations are forward-looking statements that involve a number of risks and uncertainties.Words such as "anticipates," "expects," "intends," "goals," "plans," "believes," "seeks," "estimates," "continues," "may," "will," "should," and variations of such words and similar expressions are intended to identify such forward-looking statements.Statements that refer to or are based on projections, uncertain events or assumptions also identify forward-looking statements.Many factors could affect Intel's actual results, and variances from Intel's current expectations regarding such factors could cause actual results to differ materially from those expressed in these forward-looking statements.Intel presently considers the following to be important factors that could cause actual results to differ materially from the company's expectations.Demand for Intel's products is highly variable and could differ from expectations due to factors including changes in business and economic conditions; consumer confidence or income levels; the introduction, availability and market acceptance of Intel's products, products used together with Intel products and competitors' products; competitive and pricing pressures, including actions taken by competitors; supply constraints and other disruptions affecting customers; changes in customer order patterns including order cancellations; and changes in the level of inventory at customers.Intel's gross margin percentage could vary significantly from expectations based on capacity utilization; variations in inventory valuation, including variations related to the timing of qualifying products for sale; changes in revenue levels; segment product mix; the timing and execution of the manufacturing ramp and associated costs; excess or obsolete inventory; changes in unit costs; defects or disruptions in the supply of materials or resources; and product manufacturing quality/yields.Variations in gross margin may also be caused by the timing of Intel product introductions and related expenses, including marketing expenses, and Intel's ability to respond quickly to technological developments and to introduce new products or incorporate new features into existing products, which may result in restructuring and asset impairment charges.Intel's results could be affected by adverse economic, social, political and physical/infrastructure conditions in countries where Intel, its customers or its suppliers operate, including military conflict and other security risks, natural disasters, infrastructure disruptions, health concerns and fluctuations in currency exchange rates.Results may also be affected by the formal or informal imposition by countries of new or revised export and/or import and doing-business regulations, which could be changed without prior notice.Intel operates in highly competitive industries and its operations have high costs that are either fixed or difficult to reduce in the short term.The amount, timing and execution of Intel's stock repurchase program could be affected by changes in Intel's priorities for the use of cash, such as operational spending, capital spending, acquisitions, and as a result of changes to Intel's cash flows or changes in tax laws.Product defects or errata (deviations from published specifications) may adversely impact our expenses, revenues and reputation.Intel's results could be affected by litigation or regulatory matters involving intellectual property, stockholder, consumer, antitrust, disclosure and other issues.An unfavorable ruling could include monetary damages or an injunction prohibiting Intel from manufacturing or selling one or more products, precluding particular business practices, impacting Intel's ability to design its products, or requiring other remedies such as compulsory licensing of intellectual property.Intel's results may be affected by the timing of closing of acquisitions, divestitures and other significant transactions.We completed our acquisition of Altera on December 28, 2015 and risks associated with that acquisition are described in the “Forward Looking Statements” paragraph of Intel’s press release dated June 1, 2015, which risk factors are incorporated by reference herein.A detailed discussion of these and other factors that could affect Intel's results is included in Intel's SEC filings, including the company's most recent reports on Form 10-Q, Form 10-K and earnings release.

Rev. 1/14/16 42