Semi-autonomous Teleimpedance Based on Visual Detection of Object Geometry and Material and its Relation to Environment
Georg Siegemund, Alejandro Díaz Rosales, Arne Glodde, Franz Dietrich, Luka Peternel
Delft University of Technology (Cognitive Robotics); Technical University of Berlin (Handling and Assembly Technology); CERN
IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids 2024) — IEEE · 2024

Introduction
Teleoperation allows a human operator to control a robot in complex or unstructured environments. When physical interaction is involved, however, the operator must not only position the robot but also ensure that its mechanical interaction with the environment is appropriate.
This research investigates a shared-control approach in which the operator remains responsible for motion while the robot autonomously adapts its impedance before and during interaction.
At a glance
Controlling both robot motion and interaction stiffness increases operator workload and can become risky when the properties of the remote environment are unknown.
An RGB-D vision system identifies object geometry, material, and the relationship between the object and its environment. This information is converted into an appropriate robot stiffness while the human operator controls the end-effector position.
The system successfully adapted stiffness during bolt-engagement and polishing tasks. Object and material recognition exceeded 80% precision in the reported experiments.
Shared control for physical interaction
Responsibilities are split between the human operator, who controls the desired robot position, and the autonomous system, which estimates how stiff the robot should be when interacting with the detected object.
This division reduces the number of variables the operator has to actively control during a physical interaction task.
Understanding the object before contact
Appropriate stiffness does not depend on material alone: a thin or unsupported metal plate can require gentler interaction than a thicker or better-supported object made from a different material.
The method therefore considers object identity, material, geometry, the object's relation to its environmental support, and the current interaction position.
Vision-based object estimation
RGB-D data is used for object recognition and material classification, while point-cloud processing extracts the object's geometry and its relationship to the surrounding environment.
Geometry-aware stiffness
Detected support conditions are used to determine where an object is mechanically stronger or weaker.
The autonomous controller can consequently reduce stiffness near mechanically vulnerable regions while maintaining higher stiffness where the object can safely tolerate stronger interaction.
Experimental validation
The main setup combined a KUKA LBR iiwa robot, a Force Dimension Sigma.7 haptic device, and an Intel RealSense D455 RGB-D camera. Two practical task types were investigated: engaging bolts on a plate, and polishing a stripe on a surface.
The experiments showed smooth stiffness adaptation while the operator remained in control of the end-effector position. In one comparison on an unsupported region of a metal plate, the geometry-aware method reduced bending at the free end from 6.12 mm to 2.20 mm compared with a material-only stiffness strategy.
Why it matters
Semi-autonomous impedance adaptation can reduce operator workload and improve physical interaction with remote environments, particularly when the operator has incomplete information about object geometry or mechanical support.
Highlights
- The operator commands only the end-effector position; the robot sets its own stiffness from the detected object geometry, material, and support condition.
- The estimate is made before contact, so the robot is proactive instead of reacting to forces after touching the object.
- Object and material detection reached higher than 80% precision in all performed experiments.
- On the unsupported end of a metal plate, the geometry-aware method bent the plate 2.20 mm instead of 6.12 mm for a material-only method.
Methodology
- Setup: a Force Dimension Sigma.7 haptic device commands a KUKA LBR iiwa robot, and an Intel RealSense D455 RGB-D camera near the end-effector captures the scene from different perspectives. A workspace re-indexing technique compensates for the smaller workspace of the haptic device.
- Object recognition: a YOLOv8 model trained on self-recorded, labelled images (80% training, 10% test, 10% validation) taken under different lighting conditions.
- Material recognition: a Vision Transformer trained on crops from the Materials in Context database (135,000 wood and 150,000 metal crops of 40 × 40 px). The recognised material gives a Young's modulus; together with a bending margin determined in preliminary experiments, it sets the minimum and maximum allowed stiffness.
- Geometry recognition: the RGB-D image becomes a point cloud, which is filtered by distance, clustered with DBSCAN, denoised, and segmented with RANSAC to extract position, width, length, and height. The support configuration is detected visually as the edge between object and environment.
- Impedance estimation: stiffness varies linearly with the distance from the weakest point. With support on both sides, stiffness is lowest at the centre and rises toward the supports; with one-sided support, it is lowest at the free end.
- Impedance command: a first-order integration function smooths stiffness changes so the operator is not disturbed. Damping follows the current stiffness with a damping ratio of 0.7, and the stiffness matrix is rotated into the robot base frame.
- Tasks: engaging bolts on a 500 × 300 mm plate supported on its two shorter sides, tested with wood and metal plates, and polishing a 200 × 50 mm stripe clamped on one side. The stiffness limits were found by increasing stiffness until a maximum allowed bending was exceeded and lowering it until the task could no longer be done.
Findings
- Bolt engagement: stiffness along the z-axis dropped to nearly the minimum when the operator moved near the plate centre, its weakest point, and increased toward bolts closer to the supports. The metal plate was commanded stiffer than the wooden one.
- Polishing: for the metal stripe, the maximum stiffness was set to 700 N/m (500 N/m for wood) and the minimum to 150 N/m. The z-axis stiffness was lowest at the free end and rose toward the clamp, and the x- and y-axes adapted when the operator changed the polishing direction.
- Comparison with a material-only method on a 200 × 50 × 5 mm metal plate: a material-only method would command a uniform 425 N/m, the average of the geometry-aware limits. For a reference position 20 mm below the plate, the force was 85 N at both points with that method, but 30 N at the free end and 73 N at the centre with the geometry-aware method.
- Resulting bending: at the unsupported end, 6.12 mm for the material-only method versus 2.20 mm for the geometry-aware method. At the centre the two were similar (0.67 mm versus 0.78 mm).
- Object and material detection reached higher than 80% precision in all experiments, and stiffness transitions were smooth enough not to disturb the operator's movement control.
Figures


Limitations
- Material recognition was trained on only two materials, wood and metal. It may struggle with objects made of more than one material or painted and coated surfaces.
- Stiffness is assumed to change linearly with distance from the weakest point, and the material stiffness is assumed to be uniform.
- The experiments are proof-of-concept tests on plates and stripes with a single robot setup.
- The method is deliberately low in computational cost for real-time use; more accurate estimation would cost real-time capability. Non-linear impedance estimation and combination with imitation learning are named as future work.
Citation
Siegemund, G., Díaz Rosales, A., Glodde, A., Dietrich, F., & Peternel, L. (2024). Semi-autonomous Teleimpedance Based on Visual Detection of Object Geometry and Material and its Relation to Environment. 2024 IEEE-RAS 23rd International Conference on Humanoid Robots (Humanoids), pp. 779–786. IEEE.
Interested in a research collaboration?
Get in touch to discuss joint research, data collection, or applied evaluation of new methods.