Virtual Reality Telerobotic System
Uri Kartoun, Helman Stern, Yael Edan
Department of Industrial Engineering and Management, Ben-Gurion University of the Negev, Be’er-Sheeva, ISRAEL
This paper describes a telerobotic system operated through a virtual reality (VR) interface. A least squares method is used to find the transformation mapping, from the virtual to real environments. Results revealed an average transformation error of 3mm. The system was tested for the task of planning minimum time shaking trajectories to discharge the contents of a suspicious package onto a workstation platform. Performance times to carry out the task directly through the VR interface showed rapid learning, reaching standard time (288 seconds) within 7 to 8 trials - exhibiting a learning rate of 0.79.
Teleoperation is used when a task has to be performed in a hostile, unsafe, inaccessible or remote environment (1). A telerobot is defined as a robot controlled at a distance by a human operator (HO) regardless of the degree of robot autonomy. Telerobotic devices are typically developed for situations or environments that are too dangerous, uncomfortable, limiting, repetitive, or costly for humans to perform (2). Some applications include: underwater (3), space (4), resource industry (5) and medical (6)(7).
Examples for using graphical models to allow users to control robots off-line and practice control techniques can be found in the RobotToy research (8), the KhepOnTheWeb project (9) and the WITS (Web Interface for Telescience) project (10). NASA developed WITS for controlling remote vehicles on planets such as Mars and Saturn. In the Tele-Garden project (11), users can tend a garden that contains live plants through a graphical representation of the environment. The University of Western Australia’s Telerobot experiment (12) provides Internet control of an industrial ASEA IRB-6 robot arm. The PumaPaint project (13) is a website allowing users to control a PUMA-760 robot equipped with a parallel-fingered gripper, to perform painting tasks on an easel.
The virtual reality telerobotic system, described in this paper allows a HO to: (a) perform off-line path planning by manipulating an object in a VR robotic scene, and (b) perform on-line control by indirectly controlling the real robot through manipulation of its VR representation in real-time. The available commands for control are the manipulator coordinates (x, y, z) and open / close gripper. When a command is chosen, the VR model is initially updated. After conversion, the joint angles are sent to the real robot for execution.
To demonstrate the utility of the system, we focus on the task of bag shaking. The usual method for bomb squad personnel is to blow up a suspicious bag and any explosives contained therein. However, if the bag contains chemical, biological or radiological canisters, this method can lead to disastrous results. Furthermore, the "blow-up" method also destroys important clues such as fingerprints, type of explosive, detonators and other signatures of use in subsequent forensic analysis. Extraction of the bags contents using telerobotics, which avoids these problems, is the subject addressed here. In the on-line control mode suggested here, the user controls the robot to perform the task or develops a plan off-line before downloading it to the robot’s controller for execution. For off-line bag shaking, the HO chooses a sequence of spatial locations (including inter point speeds) in the VR model to define a shaking trajectory. The trajectory is then downloaded to the real robot for execution.
In this paper we report on experiments for on-line control. The system architecture is presented in Section 2. Before carrying out the on-line control experiments it is necessary to calibrate the VR-telerobotic system. Calibration experiments are described in section 3. In section 4 we report on user experiments, using on-line control for the bag lifting and shaking task. The paper ends with conclusions and directions for future work in Section 5.
The proposed virtual reality (VR) telerobotic system contains a human operator (HO), VR web-based control interface, Internet access method, a remote server, a robot and its controller, and visual sensory feedback (Fig. 1).
The HO requires interacting with the remote robot through a control interface (Fig 2). The interface developed includes a five degree of freedom articulated robotic arm model of the "CRS A255" robot (14), two views of the real robot, a checkerboard on a table, and a world coordinate diagram that shows the x, y and z directions in the 3D scene. In addition, overall and close views of the robot site are displayed on-line.
Figure 1. System architecture layout
The system has six different operational stages controlled through predefined control panels. These are: changing the real robots speed, showing a 3D-grid (Fig. 3) that contains spatial locations which the robot gripper moves to when selected, selecting the viewing aspect of the VR model, planning shaking policies, planning off-line paths for downloading to the real robot, and on-line simultaneous control (in real-time) of the VR and real robots.
Figure 2. Web-based interface (camera views, and virtual environment)
Figure 3. 3D-grid
The virtual environment (VE) model was built and developed using "3D-Studio-Max" (15) and "Alice" (16) softwares. "Alice", a rapid prototyping software for creating interactive computer graphics applications (17)(18)(19)(20)(21), was chosen to be the VR software. It is designed to enable rapid development and prototyping of interactive graphics applications and uses "Python" (22)(23) as the language for writing its scripts.
The HO communicates with the server, connected to a robotic arm (Fig. 4) through a web-browser. Commands sent from the VE client are transmitted through TCP/IP to the server that extracts them and updates the real robot.
Figure 4. Client-server communication architecture
The remote server is connected to the robot controller, and two USB web-cameras (Fig. 1), which constantly send updated images to the client for observation of the environment. Images are in 24 bit colour and of size 240X180 pixels which appear as close-up and overall views (Fig. 2) in the client browser interface. The "A255" robot system consists of robot arm and controller. The robot arm is equipped with a special gripper capable of sliding under bags.
The inverse kinematics (IK) equations were solved using a closed form analytical solution (24)(25)(26). It has the benefit of being an exact solution and very fast to calculate. For the VR robot, an IK algorithm was implemented to determine the joint angles required to reach an end point location by supplying the (x, y, z) coordinates. The side and top views of the VR robotic chain are shown in Fig. 5. The distance x’ is obtained from the projection of the shoulder and the elbow links onto the X-Y plane. The x and the y values are the horizontal and vertical values of the robot gripper position relative to the robot base-point, PB. The z coordinate is taken vertical to the platform plane and measured from the base of L1.
(a) Side view (b) Top view
Figure 5. "A255" robotic chain
Using trigonometric identities yields:
[1]
[2]
[3]
[4]
[5]
[6] Now, given any end point position (x, y, z), the joint angles required to reach it are determined by θB, θ1, and θ2 using [1] through [6].
A transformation matrix, providing a 1 to 1 mapping between the VR and real robots is estimated from corresponding pairs of 32 intersection points on the checkerboard appearing in the VR and the real environments. The estimate is based on the method of aligning a pair of shapes, which uses a least-squares approximation (27). Given two similar shapes, x1 (the VR environment) and x2 (the real environment), a rotation θ, a scale s, and a translation (tx, ty) is calculated. The transformation equation using the 3x3 transformation matrix is shown in equation [7]. The points (xvr(i,j) , yvr(i,j)) and (xr(i,j) , yr(i,j)) are in the VR scene and in the real environments, respectively.
[7]
Given the calibrated transformation matrix an experiment to determine the transformation error was performed. The experiment (Fig. 6) starts with controlling the VR robot arm through the VR interface. The coordinates of 32 points (xvr , yvr), selected from checkerboard intersections, were set as test points. These points were inserted into [7] to obtain (xr , yr) which were sent to the real robot.
Figure 6. Calibration experiment flow chart
A pen inserted into the real robot’s gripper marked its controlled position. The coordinates of the robot’s pen on the checkerboard intersection points (xr,m , yr,m) in the real environment were measured manually by a ruler. The average transformation error between the robot’s pen positions, and the desired points in the real checkerboard was found to be 3mm.
The system was tested using on-line control through the VR interface for the task of shaking out the contents of a plastic bag. The HO views the images of the robotic workspace in the client browser, and commands the robot by selecting points in the 3D VR scene. Using the calibrated mapping of points in the VR scene to the real workspace, the robot is controlled to carry out the task. A view of the experiment to empty the contents of a plastic bag onto the platform is shown in Fig. 7. It is assumed that the bag contains ten identical electronic components known in advance. Five inexperienced operators performed ten identical experiments, and performance times (the amount of time it takes to extract all the objects from the bag) were recorded. The learning curve of task completion time was reduced quickly reaching standard time (288 seconds) after 7 to 8 trials (Fig. 8).
(a) Overall view
(b) Close view
Figure 7. "A255" robot, plastic bag, platform and electronic
components
Figure 8. Learning curve experimental results
This paper describes the design, implementation
and testing of a real-time VR-telerobotic web based system. Visual feedback
arrived to the human interface via two independent web-cameras and a transformation
matrix mapped the VE to the real one was calculated to calibrate the system.
The system allows a human operator to: (a) perform off-line path planning by
manipulating an object in a VR robotic scene, (b) perform on-line control by
indirectly controlling the real robot through manipulation of its VR representation
in real-time. A least squares method is used to find the orientation, rotation
and translation of 1 to 1 mapping between the virtual and real environments.
Results revealed an average transformation error of 3mm. The system was tested
for the task of planning minimum time shaking trajectories to discharge the
contents of a suspicious package onto the workstation platform. Performance
times to carry out the task directly through the VR interface showed rapid learning,
reaching standard time (288 seconds) within 7 to 8 trials. This fast learning
rate was calculated as 0.79. A future extension is to use visual feedback from
the cameras for creating a second transformation mapping the real environment
to the virtual one. Errors for both transformations will be calculated and the
real robot will perform corrections in order to reduce it. Possible extension
of the VR graphical display of the robotic environment presented here is to
use visual input (e.g., data gloves, movement trackers, hand gestures)
and output (e.g., head mounted displays, shutter glasses) devices. Future
research will include development of a cooperative human-robot learning system
for remote robotic operations using a VR interface (28).
This project was partially supported by the Ministry of Defense MAFAT Grant No. 1102, and the Paul Ivanier Center for Robotics Research and Production Management, Ben-Gurion University of the Negev.
(1) J. Bukchin, R. Luquer, and A. Shtub, "Learning in tele-operations", IIE Transactions, vol. 34, no. 3, pp. 245-252, 2002.
(2) N. Durlach and S. N. Mavor, Virtual Reality: Scientific and Technological Challenges, National Academy Press, Washington D.C., 1995.
(3) L. Hsu, R. Costa, F. Lizarralde, and J. Soares, "Passive arm based dynamic positioning system for remotely operated underwater vehicles", IEEE International Conference on Robotics and Automation, vol. 1, pp. 407-412, 1999.
(4) G. Hirzinger, B. Brunner, J. Dietrich, and J. Heindl, "Sensor-based space robotics-ROTEX and its telerobotic features", IEEE Transactions on Robotics and Automation, vol. 9, no. 5, pp. 649-663, 1993.
(5) A. A. Goldenberg, J. Wiercienski, P. Kuzan, C. Szymczyk, R.G. Fenton, and B. Shaver, "A remote manipulator for forestry operation", IEEE Transactions on Robotics and Automation, vol. 11, no. 2, pp. 185-197, 1995.
(6) D. Kwon, K. Y. Woo, and H. S. Cho, "Haptic control of the master hand controller for a microsurgical telerobot system", IEEE International Conference on Robotics and Automation, pp. 1722-1727, 1999.
(7) D. Sorid and S. K. Moore, "The virtual surgeon", IEEE Spectrum, pp. 26-39, 2000.
(8) http://robotoy.elec.uow.edu.au/
(9) O. Michel, P. Saucy, and F. Mondada, "KhepOnTheWeb: an experimental demonstrator in telerobotics and virtual reality", Proceedings of the International Conference on Virtual Systems and Multimedia, IEEE VSMM’97, pp. 90-98, 1997.
(10) P. G. Backes, K. S. Tso, and G. K. Tharp, "The web interface for telescience", Presence, MIT Press, vol. 8, pp. 531-539, 1999.
(11) K. Goldberg, J. Santarromana, G. Bekey, S. Gentner, R. Morris, J. Wiegley, and E. Berger, "The Telegarden", Proceedings of ACM SIGGRAPH, 1995.
(12) K. Talyor and J. Trevelyan, "A telerobot on the world wide web", National Conference of the Australian Robot Association, 1995.
(13) M. Stein, "Interactive internet artistry, painting on the world wide web with the PumaPaint project", IEEE Robotics and Automation Magazine, vol. 7, no. 1, pp. 28-32, 2000.
(14) "A255" Robot System, CRS Robotics Human Scale Solutions, 1998.
(15) http://www.discreet.com/
(16) http://www.alice.org/
(17) M. J. Conway, R. Pausch, R. Gossweiler, and T. Burnette, "Alice: a rapid prototyping system for building virtual environments", Adjunct Proceedings of ACM CHI’94 Human Factors in Computing Systems Conference, vol. 2, pp. 295-296, 1994.
(18) R. Pausch, T. Burnette, A. C. Capehart, M. Conway, D. Cosgrove, R. DeLine, J. Durbin, R. Gossweiler, S. Koga, and J. White, "A brief architectural overview of Alice, a rapid prototyping system for virtual reality", IEEE Computer Graphics and Applications, 1995.
(19) M. J. Conway, "Alice: easy-to-learn 3D scripting for Novices", Ph.D. Thesis, Faculty of the School of Engineering and Applied Science at the University of Virginia, 1997.
(20) S. Cooper, W. Dann, and R. Pausch, "Alice: a 3-D tool for introductory programming concepts", Proceedings of the Fifth Anual CCSC Northeastern Conference, pp. 107-116, 2000.
(21) W. Dann, S. Cooper, and R. Pausch, "Making the connection: programming with animated small world", Fifth Annual SIGCSE/SIGCUE Conference on Innovation and Technology in Computer Science Education (ITiCSE), pp. 41-44, 2000.
(22) http://www.python.org/
(23) M. Lutz, Programming Python, O’Reilly and Associates, 2001.
(24) J. Lander, "Oh my god, I inverted kine", Game Developer Magazine, pp. 9-14, 1998.
(25) J. J. Craig, Introduction to Robotics: Mechanics and Control, Addison-Wesley, 1989.
(26) P. J. McKerrow, Introduction to Robotics, Addison-Wesley, 1991.
(27) T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, "Training models of shape from sets of examples", Proceedings of the British Machine Vision Conference, pp 9-18, 1992.
(28) U. Kartoun, "A human-robot collaborative learning system using a virtual reality telerobotic interface", Ph.D. Thesis Proposal, Department of Industrial Engineering and Management at the Ben-Gurion University of the Negev, Israel, 2003.