Semi-autonomous Teleimpedance Based on Visual Detection of Object Geometry and Material and its Relation to Environment
Georg Siegemund, Alejandro Díaz Rosales, Arne Glodde, Franz Dietrich, Luka Peternel
IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids 2024) — IEEE · 2024
Introduction
Teleoperation allows a human operator to control a robot in complex or unstructured environments. When physical interaction is involved, however, the operator must not only position the robot but also ensure that its mechanical interaction with the environment is appropriate.
This research investigates a shared-control approach in which the operator remains responsible for motion while the robot autonomously adapts its impedance before and during interaction.
At a glance
Controlling both robot motion and interaction stiffness increases operator workload and can become risky when the properties of the remote environment are unknown.
An RGB-D vision system identifies object geometry, material, and the relationship between the object and its environment. This information is converted into an appropriate robot stiffness while the human operator controls the end-effector position.
The system successfully adapted stiffness during bolt-engagement and polishing tasks. Object and material recognition exceeded 80% precision in the reported experiments.
Shared control for physical interaction
Responsibilities are split between the human operator, who controls the desired robot position, and the autonomous system, which estimates how stiff the robot should be when interacting with the detected object.
This division reduces the number of variables the operator has to actively control during a physical interaction task.
Understanding the object before contact
Appropriate stiffness does not depend on material alone: a thin or unsupported metal plate can require gentler interaction than a thicker or better-supported object made from a different material.
The method therefore considers object identity, material, geometry, the object's relation to its environmental support, and the current interaction position.
Vision-based object estimation
RGB-D data is used for object recognition and material classification, while point-cloud processing extracts the object's geometry and its relationship to the surrounding environment.
Geometry-aware stiffness
Detected support conditions are used to determine where an object is mechanically stronger or weaker.
The autonomous controller can consequently reduce stiffness near mechanically vulnerable regions while maintaining higher stiffness where the object can safely tolerate stronger interaction.
Experimental validation
The main setup combined a KUKA LBR iiwa robot, a Force Dimension Sigma.7 haptic device, and an Intel RealSense D455 RGB-D camera. Two practical task types were investigated: engaging bolts on a plate, and polishing a stripe on a surface.
The experiments showed smooth stiffness adaptation while the operator remained in control of the end-effector position. In one comparison on an unsupported region of a metal plate, the geometry-aware method reduced bending from 4.32 mm to 2.20 mm compared with a material-only stiffness strategy.
Why it matters
Semi-autonomous impedance adaptation can reduce operator workload and improve physical interaction with remote environments, particularly when the operator has incomplete information about object geometry or mechanical support.
Figures
Citation
Siegemund, G., Díaz Rosales, A., Glodde, A., Dietrich, F., & Peternel, L. (2024). Semi-autonomous Teleimpedance Based on Visual Detection of Object Geometry and Material and its Relation to Environment. 2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids), pp. 779–786. IEEE.
Interested in a research collaboration?
Get in touch to discuss joint research, data collection, or applied evaluation of new methods.