2005 IEEE Workshop on Advanced Robotics and its Social Impacts

Internet Based Robots: Applications, Impacts, Challenges and Future Directions Amit Goradia and Ning Xi

Imad H. Elhaj

Dept. of Electrical and Coinputer Engineering Michigan State University

Dept. of Computer Science and Engineering Oakland University Rochestel; MI 48309, USA elhajj@ oakSand.edu

East Lansing, MI 48824, USA xin@egi".edu

Abstract-The Internet has ushered in the era of an exceedingly connected digital lifestyle. It has brought about a lot of changes in the social arena and has revolutionized the way we communicate with each other. Internet based teleoperation systems utilize the ubiquitous connectivity and low cost bandwidth offered by the Internet to send commands and receive supermedia feedback for teleoperating remote systems. TeIeoperation systems have many myriad applications and impact areas ranging from education and entertainment to remote exploration and medicine. Such wide ranging applications of Internet based teleoperation systems will have many social impIications and a profound impact on the day-to-day human lifestyle. This paper tries to elaborate on the many application areas of Internet based teleoperation and further discusses the social implications of the same. Further the paper discusses various technical challenges and research issues related to Internet based teleoperation systems and provides a few experimental results on some teleoperation systems implemented at the Robotics and Automation Laboratory at Michigan State University and the Communication and Robotics Laboratory at Oakland University. Index Terms-Teleoperation, Supermedia, Overlay Networks, Internet.

and actions in the virtual domain. Teleoperation of remote systems will enable the virtual world to reach out to the real one. Using teleoperated robots, humans can forge bond from the virtual world to the real world. This will have a profound impact on the way humans carry out their day to day activities. The terms telepresence [l] or remote presence are used to describe the experience of being fully present at a live real world location remote from one's own physical location. The collection of all feedback streams, such as video, audio, haptic temperature and any other type of perception that can be quantified, transmitted and reproduced for the operator in an effort to obtain as much information as possible about the current state of the robot and its surroundings can be termed as supermedia. Aided by supermedia feedback and the telepresence experience the operator can behave, receive stimuli and react as though they were at the remote site.

I. INTRODUCTION Teleoperation implies control of a system at a remote site generally inaccessible by the operator. The operator and the remote system are physically separated and must interact to accomplish the task. The concept of teleoperation systems has been around for decades. A teleoperation system generally consists of the following basic components: remote system with sensors, actuators, controllers and power systems, a local interface which allows the operator to generate commands and receive sensory feedback from the remote environment and a communication medium through which the information is exchanged between the local and remote environments. Traditionally, teleoperation systems used a dedicated communication channel which incurs high cost and inflexibility. However the growth of the Internet opens a new stage for teleoperated systems. The Internet has added a new dimension to the day-today human lifestyle. Internet based applications and services are continuously expanding to cover many aspects of our lives. A person can now trade, Iearn, play, observe and do many other activities over the Internet. Yet the Internet is still largely a passive medium which aids observation 0-7803-8947-6/05/$20.00@2005 IEEE

Hong Kong

Fig. 1. Basic Idea of Internet based teleoperation systems.

Figure 1 shows the basic idea of the supermedia enhanced Internet based teleoperation system . Teleoperation aided by telepresence will enable humans to extend the boundaries of they physical world and interact with each other through physical avatars. This concept of many people in different physical locations meeting in a common physical location through the use of avatars can be seen as the analog of chat rooms on the Internet which are so 73

prevalent as a forum for social interaction.

A. Motivations The desire to reach out and affect a remote environment has driven research in the field of teleoperation for decades. Teleoperation, in general, can be carried out using any kind of communication media. But since the Internet has matured over the past decade into its present state, interest has been growing in using the Internet for teleoperation. The Internet opens up the possibility of ubiquitous access all over the world. It is less expensive (free in many cases), more publicly available and does not require specialized (and expensive) hardware when compared to a dedicated communication link. The Internet has opened up many avenues for information transfer and communication and has facilitated information access to many remote comers of the globe. It thus enables a truly ubiquitous and global medium which can be used to reach out and affect remote environments. It enhances reachability and satiates the desire for remote interaction and telepresence. Teleoperation can also be harnessed for facilitating cooperation and coordination between humans physically separated from one another. The teleoperated robots can be construed as an avatar of the operator and can observe and interact with other such avatars and humans present in the remote environment. This opens up the possibility of remote social interaction and has the potential to revolutionize the way we think about human contact, interaction and relationships. B. Inreiner Bused Robots Internet-based robots can be divided into three categories according to the method used to send commands and receive feedback and according to the nature of feedback supptied. These three categories are referred to here as: teleprogrammed, telesimulated, and real real-time teleoperated. Teleprogrammed Internet-based robots are ones that require the operator to upload a plan or set of commands for it to execute. This uploaded program is executed by the robot either autonomously or semi-autonomously. So this category can be further divided into ”program and watch” robots or ”program, watch and intervene” robots. Program and watch robots are ones that execute the program without allowing the operator to change the commands on the fly. Program, watch and intervene robots have the added option of the operator changing the program on the fly before the robot finishes executing it. Typically, teleprogrammed robotic systems include only visual feedback in the form of video. Telesimulated Internet-based robots include systems that feed forward commands in real-time but the feedback in simulated. So the operator sends commands in real-time however the feedback supplied to the operator is simulated. This simulated or predicted feedback can be one of two forms; either its completely simulated based on a robot and environment model, or its partially simulated based on

a model and corrected by actual feedback. Generally feedback is in the form of predicted display andor predicted force. Real real-time teleoperated Internet-based robotic systems feed forward commands in real-time and feedback real real-time sensory information. The feedback comes in several forms the most typical of which are video and force.

11. APPLTCATIONS AND IMPACT AREAS The major factors in determining the social and economic implications of Internet based teleoperation will be the applications which will utilize such systems. The application areas can be divided into consumer, commercial and industrial and research markets. Consumer markets primarily involve home users and the application areas would include home control, entertainment and education. For example the ability to remotely and interactively experience live events and games or interactively study and learn about remote locations of interest. Education would benefit from the ability of students to remotely access laboratory equipment. Commercial markets involve a broader range of applications than the consumer markets. Applications in the service sector, tourism, commerce, manufacturing and maintenance industries can involve tele-services, telemedicine, entertainment, remote factory maintenance etc. Security, law-enforcement and defense applications also have tremendous potential. Applications of Internet based teleoperation in industrial and research environments could involve remote underwater or deep-space exploration, hazardous material handling, remote search and rescue operations etc. A. Remote Exploration

This application became very popular during the NASA Mars Pathfinder Mission [ 2 ] . There are various obvious advantages for this application and the importance of teleoperation for space and deep sea exploration is clear, especially in the safety and cost reduction aspects 131 141. However, due to the large delays in space applications real real-time teleoperation is not feasible with the current technology. E. Medicine

Despite all the skeptics, this application is getting major attention from the research community and is experiencing significant advances [ 5 ] - [6]. The ultimate goal is to have a highly reliable and accurate system that can be trusted with human life. This will allow experts to diagnose, treat and to operate on patients who are thousands of miles away. Uses range from emergency trauma care in ambulances and other emergency services, home care involving remote assistance for elderly and disabled people to physical. therapy and even specialized surgery. Although this is one of the major driving applications it is expected to be the last to adopt this technology over non-dedicated communication lines because of the many safety concerns.

74

C. Entertabiment Several teleoperated games are found on the Internet; for example, in Germany there is an interactive railroad that can be operated remotely. The entertainment industry was one of the first to notice the importance of networking thus strived to link players all over the world with networked games. Now there is the possibility to link several players with several machines or robots. Telepresence and the ability to actively affect and perceive the remote environment opens the liood gates to many travel related industries. For example a travel agent could let potential customers experience a remote holiday location and also interact with olher people there to inquire about their experience. Various similar industries such as rea1 estate, museums and live events could make active use of teleoperated devices.

D. Services Machines and complete factories can be monitored from hundreds of miles away. These can also be maintained and controlled remotely. People can attend remote conferences and meetings without actually having to go there through physical avatars. Further they can interact and coordinate with other people and avatars at the remote site. Experts in a particular field can lend a helping hand at remote site without having to physically be present.

E. Hazardous Material Handling One of the first noticed applications of teleoperation is handling hazardous material; such as, radioactive, chemical or explosive substances [7]. For example, tekoperation can he a natural answer for chemical spill cleanup. It can also be used to alleviate the need to expose employees to unpleasant and potentially dangerous working conditions like mining and bomb disposal. Currently, this is one of the main applications that got out of the laboratory setup and into the field. E Law Enforcement

Several law enforcement agencies already own robots that can be teleoperated to replace humans in hazardous situations and environments.

G. Search and Rescue Services MultjpIe robots can be used to cover unaccessible areas in search and recovery efforts. Several robots were used for assistance in the recovery efforts at “Ground Zero” shortly after the World Trade Center catastrophe [SI. H. Education Several robots are used online for educational purposes whether in a formal or informal setup [9]. For example a student sitting in a classroom or at home will be able to study and experience a wide variety subjects and phenomena from all over the world like the habitat of wild animals and deep sea creatures to museums located all over the world.

111. SOCIALIMPACTS The previous section outlines many applications for Internet based teleoperation systems. Judging hy these applications, the social implications can be quite profound. The most obvious implication would be the reduction in personal travel for business and work related purposes. The ability to project oneself to a remote location will reduce the need to travel to remote destinations. Similar implications have been seen with the advent of the mobile office whereby a person can work and communicate from a remote location without actually having to go to a cubicle. It will reduce the travel expenses for company executives and result in cost savings for the company. However, this reduction in travel will affect the airline and hotel industry. Reduction in personal travel will also aid in reducing the pollution caused by the road and air travel which is definitely a positive impact on society. Ensuring the safety and health of employees by obviating the need to work in hazardous environments and unpleasant working conditions while still Ietting them perform their jobs in an effective manner is a basic advantage of teleoperation technology. Mining, handling hazardous material and other such high risk jobs can be performed remotely without ever having to be exposed to the inherent risk involved in performing these jobs. Further law enforcement and battlefield operations can also he conducted remotely thus ensuring the safety of the soldiers and law enforcement officials. Rescue missions for disasters can also be conducted remotely without risking the lives of the fire fighters and other rescue workers. Students can interact with teachers and other students from a different part of the world, This kind of interaction exposes them to various cultures and ushers in a greater global understanding and tolerant outlook, Currently there are study abroad programs at various universities which fulfil this need to a more global outlook. However they are expensive and expose the students to only a single culture. Laboratories, museums, heritage centers etc, can offer remote tours to the public using Internet based teleoperation systems whereby people all over the world can log on and enjoy the interactive experience. The advent of the Internet brought about a new means to communicate using e-mails, chat rooms, instant messaging, SMS and many such personalized communication services. These new methods of communication have revolutionized the way we interact with one another. Today there are even online published conferences which can be attended remotely by the delegates who can passively observe the presentations made at remote locations. Internet based telepresence systems can enable the delegates at a conference or members of an interest group located all over the world to interact physically with one another without the need for travel to a common physical location. The Internet has also brought about the concept of the global market place where people can hid, buy, sell and conduct business on a day to day basis. However current systems are not physically interactive and do not provide a 75

real immersible experience of a real market place or brick and mortar store. Interactive teleoperation can provide the ability to touch feel and test the products before actually buying them. Tele-cooperation and coordination can help people located in many remote locations to work together towards a common goal. However, no matter how developed the technology becomes teleoperation will never become equivalent to the real experience. In addition, the human touch and interaction would be lost in the face of the robot. Instead of a human assisting an elderly now the machine does this. The phycological and social impacts of teleoperation are yet to be formally studied. Although economically teleoperation seems to be the right thing but as is the case with any new technology and paradigm shifts the social effect must be considered.

Iv. TECHNICAL CHALLENGES AND RESEARCHISSUES For teleoperation systems to be widely acceptable, several safety and performance specifications have to be met. Stability, transparency and synchronization are among the main features that are desired in a teleoperation system. The teleoperation system should be stable under all operating conditions. Transparency relates to the concept of telepresence and provides a measure of the realism of the operation. The operator and the remote system should be synchronized irrespective of the communication medium used. Further the various supermedia streams should also be synchronized as the telepresence experience depends on it. However, the synchronization is not with respect to time as that is not possible due to delays. Synchronization referred lo here is with respect to the state of the system. For example, if the operator believes the robot is in a specific location then it is guaranteed that the robot is within a bounded error of that location. In addition reliability, safety and efficiency issues also need to be addressed. However ensuring these features for Internet based teleoperation systems is challenging. The main difficulties relate to the lack of a general design method for such systems. Also the human operator within the control loop adds to the complexity of system design and analysis process. In addition, teleoperation systems are characterized by non-determinism in the environment, which is a limitation in case of certain assumptions. However, the main difficulties are caused by the characteristics of the network and communication medium especially in the case of the Internet.

A. Intemes and related issues As any communication link, the Internet introduces delay into the system. This delay can be due to several reasons: propagation, congestion, processing, or lack of resources. The biggest challenge of Internet based teleoperation is making the unpredictability caused by the Internet to be transparent to the control loop of the system. Random time delay, packet loss, network buffering effects and disconnections in the Internet’s best-effort service model

present major difficulties. Among all the uncertainties, time delay is one of the biggest obstacles to build a stable teleoperation control system. High latency may cause the robot to stall in the middle of a task or the operator to lose control of the remote robot. The uncertain and nondeterministic variations of the network latency makes the control system unstable, which in turn affects of the performance of the teleoperation application. This problem may be approached through two directions: 1) Try to modify the control mechanism of the teleoperation system to accommodate the unpredictable nature of the Internet. The traditional control system assumes a dedicated and reliable communication channel between the operator and the robot. In order for the system to work over the Internet, some of its assumptions must be changed. 2) Try to improve the quality of service of the Internet to make it dose to the dedicated communication channels used by the traditional system. One of the possible ‘ways to help build a stable control system is to reduce the end-to-end latency (or its variance) of the communication channels. Event based teleoperation systems [ 101, E1 I ] were presented to address the problem from the first direction. The performance difficulty caused by the Internet based teleoperation system is a result of using time as a reference for different system entities. In the event-based control approach, a non-time based reference is used. An event reference is a monotonically increasing parameter to synchronize the operator and the robot. Through the event reference, low level sensing and control modules are integrated with high level task scheduling and action planning. The system is able to cope with the uncertainties caused by the Internet and provide event transparency to the system user.

B. Human / Machine interj4ace

From the point of view of an operator’s experience, teleoperation and the telepresence experience is far from being close to direct operation. This is due to the limited sensing capabilities of the remote operator which tends to disconnect the operator from the environment and the task. It has been shown that the operators performance during a teleoperation task is improved when haptic feedback from the remote environment alongside video and audio feedback are provided [12]. Force is the most common form of haptic feedback used for teleoperation tasks. However, there are many applications which require additional types of feedback for feasibility and safety. An example is classification and identification where temperature is required to identify certain objects and materials. Also in some medical applications the temperature change of the organ being operated on is critical for safety. Supermedia can be defined as a collection of all the feedback streams, such as haptic, video, audio, temperature and any other type of perception that can be quantified, 76

M F Fn-

Wireless Clhcmel

(Law

Fig. 2.

Sonar h f r d

Cameras Tcmperatiue)

Tempemlure Rmdenng Dcnce

.

The hardware architecture of the teleoperation systems developed.

between both is the physical contact an influence. If the robots get in contact with each other or the same object at the same time it is said to be coordination; otherwise, it is collaboration. A tele-autonomous robotic formation system where an operator is able remotely to alter the formation movement in the case of an unexpected event. A micro manipulator teleoperation system was implemented. The micro manipulator can be controlled by sending position increments and force is sent back from a one dimensional micro force sensor.

transmitted and reproduced for the operator in an effort to obtain as much information as possible about the current state of the robot and its surroundings. C. Cooperation and Coordination

Cooperating robots have been the topic of several research efforts ranging from autonomous manipulation to remote control [13], [14]. In many scenarios, cooperation offers additional safety, efficiency and feasibility. For instance, an object might be too heavy or too fragile to be lifted by one robot only. Combined with the Internet, tele-cooperation provides the means for multi-experts at multi-sites to cooperate using multi-robots. Coordinating robots and more specifically telecoordinating robots, where the robots are not autonomous but teleoperated, offer numerous technical difficuIties. These difficulties relate to the synchronization of operation, coordination strategies and constraints on the manipulated

All these systems were controlled bilaterally via the Internet in real-time. The performance illustrated that these systems are immune to delay effects. This is not to say that delay was eliminated or reduced but it did not have stability of operation of the teleoperation system. VI. CONCLUSIONS

object.

V. EXPERIMENTAL IMPLEMENTATION At Michigan State University in cooperation with Oakland University, The Chinese University of Hongkong and Nagoya University in Japan, we have carried out extensive research into Intemet based teleoperation systems. The general system hardware architecture is shown in figure

2. To illustrate the wide range of possibilities with teleoperation, severa1 different teleoperation systems were implemented and tested. These teleoperation systems included: A mobile robot teleoperated by sending velocity commands and feeding back virtual force. This force corresponded to the distance to obstacles in the direction of motion. A mobile manipulator was teleoperated by sending either position or velocity commands and feeding back force from the forcekorque sensor mounded on the manipulator. Multi-site multi-operator teIe-cooperation and telecoordination systems where implemented where remote operations control distinct robots for the purpose of collaboration or coordination. The difference

Internet based teleoperation systems are poised to grow at a very fast pace and have the ability to offer myriad services ranging from remote exploration and search and rescue to medicine and manufacturing services. The widespread use of Internet based teleoperation technology is bound to have many economic and social implications and has the potential to revolutionize the way we interact with other human beings. However the field is still in a nascent stage and more research needs to be done in the fields of Internet based realtime communication, human machine interaction, tele-cooperation and tele-coordination to take advantage of all its merits. This paper has presented a general overview of some application areas and the social implications of widespread adoption of the technology. The paper also describes briefly some of the experiments conducted at Michigan State University and Oakland University on Internet based teleoperation. REFERENCES

[l] G.M. Mak, “Telepresence-the technology and its economic and social implications;’ in Technology and Socieiy, 1497. ‘Technology and socieiy ut a 7ime of Sweeping Change’. Proceedings., I997 Infernatiaml Symposium on, 1997, pp. 118 124. ~

77

[2]

K. S. Tu, P. G. Backes and G. K. Tharp, “Mars Pathfinder Mission Internet-Based Operations Using WITS,” in Workshop on Inreinet Robotics IEEE International Coilference on Robotics and

Automution, 1999, pp. 284-291. [3] E Lizarralde J. Soares L. Hsu, R. Costa, “Passive Arm Based Dynamic Positioning System for Remotely Operated Underwater Vehicles,” in IEEE International Conference un Robotics and Auromotion, 1999, vol. 1, pp. 407-412. 141 J. Dietrich G. Hirzinger, B. B m n e r and J. Heindl, “Sensor-Based Space Robotics-ROTEX and Its Telerobolic Features,” vol. 9, no. S, October 1993. [5] H. S Cho D. Kwon, K. Y. Woo, ‘Waptic Control of the Master Hand Controller for a Microsurgical Telerobot System,” in IEEE International Conference on Robotics and Automation, 1999, vol. 3, pp. pp. 1722-1727. [6] Y. Wang M.Ghodoussi, S. Bntner, “Robotic Surgery -The Transatlantic Case,” in IEEE International Conference on Robotics and Automation, 2002. [7] R. V. Dubey S. E. Everett, “Model-Based Variable Position Mapping for Telerobotic Assistance in a Cylindrical EnvironmenG” in IEEE International Conference on Robotics and Automation, 1999, vol. 3, pp. 2197-2202. [8] Robin Murphy, “Human-Robot Interaction in Robot-Assisted Urban Search and Rescue,” in Workshop on Human-Robot bzreraclion IEEE hirernafioizal Conference on Roborics and Aululnafiora, 2002. [9] Gerard McKee, “The development of Internet-Based Laboratory Environments for Teaching Robotics and Artificial Intelligence,” in lnieriiaiional CortJerence on Robotics and Automation, 2002. [IO] lmad Elhaji Ning Xi Wai-bung Fung Yun-€hi Liu Y. Hasqawa Toshio Fukuda, “Supermedia-enhanced Internet-based telerobotics,” Proceedings of the IEEE, vol. 91, no. 3, pp. 396461,2003. [ l l ] Ning Xi and T, J. Tam, “Action Synchronization and Control of Internet Based Telerobotic Systems,” in I€€€ lnienialional Conference on Robotics and Autoinofion, 1999. [I21 S.A. Wall and W.S. Hanvin,“Quantification of the effects of haptic feedback during motor skills tasks in a simulated environment,” in Proceedings of ihe 2nd PHANTOM Users Research Synposiurn, 2000. pp. 61-69. [13] h a d Elhajj Amit Goradia Ning Xi Chow Man Kit Yun Hui Liu Toshio Pukuda, “Tele-caordina~ed Control of Multi-robot Systems via the Internet,’’ in hirernational Conference on Robotics and Auromnriorr, 2003.

[ 141 Jindong Tan Ning Xi Amit Goradia Weihua Sheng, “Coordination of Human and Mobile Manipulator Formation in a Perceptive Reference Frame,” in Intematiorml Conference O R Robotics and

Automotion, 2003.

78