A Whole-Body Integrated AVATAR System: Implementation of Telepresence with Intuitive Control and Immersive Feedback

Abstract

This article proposes an intuitive and immersive whole-body teleoperation system with motion-based control and multimodal feedback. The system consists of an anthropomorphic teleoperated robot and a haptic interface platform. The teleoperated robot has dual arms with dexterous hands, a head with a neck, and waist, giving it a human-like appearance and wide range of motion as well as an omnidirectional mobile platform for improved mobility. The haptic interface platform enables a human operator to control the robot intuitively by measuring the operator’s motion with a motion-capture system, providing haptic feedback to the user’s arms, fingers, and feet, and providing 3D image feedback. Additionally, facial animation further enhances immersion by synchronizing the facial expression of the robot with the user’s voice. The proposed teleoperation system offers a promising solution for the human-oriented robotic avatar system, which was verified through a global competition: the $10 M All Nippon Airways (ANA) Avatar XPRIZE. The system was successfully evaluated with 45 min of training time for users who were new to our system. And the lessons learned from the competition and future improvements are discussed.