Fisher, Bob

Khadem, Mohsen

Zhaole, Sun

Institute of Perception, Action and Behaviour (IPAB), School of Informatics, University of Edinburgh

2025-09-17T11:27:09Z

2025-09-17T11:27:09Z

2025-09-17

Humans use their hands to dexterously manipulate cables to perform various tasks, like grasping cables, moving cables in hand without dropping them, bending the cable into a U shape for hooking, USB-cable insertion and so on. Unlike dexterous manipulation of rigid objects, dexterous cable manipulation skills are still underexplored in robotics due to the unique challenges posed by cables' deformability and uncertainty. Thus, we focused on using a multi-fingered hand to perform dexterous manipulation of cables. To build a robotic system for dexterous cable manipulation, we first need a good perception of the cable. During manipulation, the cable is often partially occluded by fingers. Thus, our first work is to propose a robust cable perception pipeline in 3D against occlusions. We followed three steps: first, extracting a 2D mask from an RGB image, getting several key points to describe the target cable from the segmented point cloud, and finally, applying physical smoothing to the key points to make the reconstructed cable physically realistic. After establishing a good cable perception system, we focused on the multi-fingered hand, a new end-effector for cable manipulation. While existing research has addressed cable manipulation with grippers, using a dexterous hand introduces specific difficulties in tasks such as cable grasping, sliding, in-hand bending, etc, for which no dedicated task definitions, benchmarks, or success metrics exist. Our initial exploration was based on using a multi-fingered hand-in simulation to perform some basic cable manipulations. Due to the difficulty of high-dimension control and limited hand motion data, we used Reinforcement Learning to train an agent based on an anthropomorphic hand with 20 degrees of freedom. We proposed five tasks, including in-hand cable sliding from left to right and from right to left, object lifting, cable end-tip position control, and cable bending. We achieved relatively good results with over 60% success rate on these tasks. However, implementing the system in the real world becomes an entirely different case, and three things are needed: 1. a benchmark for dexterous cable manipulation, 2. a better-designed multi-fingered hand for dexterous cable manipulation, and 3. a controller that can perform high-dimensional manipulation in the real world. We provided three solutions: (1) We first defined and concluded a series of dexterous cable manipulation tasks into a taxonomy covering most one-hand cable manipulation short-horizon primitives and long-horizon tasks. This proposed taxonomy revealed that thumb-index composition is critical for cable manipulation and decomposed long-horizon tasks into shorter primitives. (2) We designed a new five-fingered hand with 25 degrees of freedom. It has two symmetric thumb-index compositions and a rotatable joint on each fingertip, which allows it to perform tasks that are even difficult for humans. Besides, we created a demonstration data collection pipeline for this hand. (3) We defined a finite state machine based on collected demonstrations of short-term primitives. The hand can robustly replay 7 short-horizon primitives with over 90% success rate on cables of the same material and over 72% on cables of different materials. For long-horizon tasks which need combinations of at least three primitives, the hand can replay four long-horizon tasks with over 75% on cables of the same material.

https://hdl.handle.net/1842/43974

http://dx.doi.org/10.7488/era/6504

en

The University of Edinburgh

Sun Zhaole, Hang Zhou, Li Nanbo, Longfei Chen, Jihong Zhu, and Robert B. Fisher. “A Robust Deformable Linear Object Perception Pipeline in 3D: From Segmentation to Reconstruction” IEEE Robotics and Automation Letter, 2024

Sun Zhaole, Jihong Zhu, and Robert B. Fisher. “DexDLO: Learning Goal- Conditioned Dexterous Policy for Dynamic Manipulation of Deformable Linear Objects.” IEEE International Conference on Robotics and Automation, 2024

Sun Zhaole, Jihong Zhu, and Robert B. Fisher. “Leverage-Suction Hand: A Multi-fingered Hand with Suction Cups on Fingertips for Multi-Object Grasping and Manipulation.” IEEE International Conference on Robotics and Automation, Workshop on Multi-Object Grasping: Progress and Prospects, 2024

2026-09-17

Dexterous Manipulation

Robotic Perception

Robotic dexterous manipulation of cables

Thesis or Dissertation

Doctoral

PhD Doctor of Philosophy

RESTRICTED ACCESS