RealityBrush: an AR authoring system that captures and utilizes kinetic properties of everyday objects

Abstract

This study introduces RealityBrush, a novel augmented reality (AR) authoring system that allows designers to quickly and easily create realistic virtual objects by capturing and utilizing the kinetic properties of everyday physical objects in the early stages of design. The RealityBrush system consists of a handheld device, a data analysis module and an AR feedback module. The handheld device, which is made in the shape of a rod, is equipped with a depth camera and a force sensor at the tip. When a user holds the device and pokes a physical object, the local force applied to the object and the resulting deformations of the object are measured simultaneously. By analyzing the relationship between measured force and deformations, the RealityBrush system can identify two kinetic properties of the poked object: stiffness and motion resistance. The user can then use the handheld device as a 3D brush to create a virtual object in the air and assign the measured kinetic properties to the created virtual object. Finally, the system's physics engine allows the user to interact with the created object by using the device to poke or push the object. The technical evaluation showed that the system can successfully extract the stiffness and motion resistance of everyday objects. We also report initial user feedback on AR authoring using the RealityBrush system.