Robotic dexterous manipulation of cables

Abstract

Humans use their hands to dexterously manipulate cables to perform various tasks, like grasping cables, moving cables in hand without dropping them, bending the cable into a U shape for hooking, USB-cable insertion and so on.

Unlike dexterous manipulation of rigid objects, dexterous cable manipulation skills are still underexplored in robotics due to the unique challenges posed by cables' deformability and uncertainty. Thus, we focused on using a multi-fingered hand to perform dexterous manipulation of cables.

To build a robotic system for dexterous cable manipulation, we first need a good perception of the cable.

During manipulation, the cable is often partially occluded by fingers. Thus, our first work is to propose a robust cable perception pipeline in 3D against occlusions.

We followed three steps: first, extracting a 2D mask from an RGB image, getting several key points to describe the target cable from the segmented point cloud, and finally, applying physical smoothing to the key points to make the reconstructed cable physically realistic.

After establishing a good cable perception system, we focused on the multi-fingered hand, a new end-effector for cable manipulation.

While existing research has addressed cable manipulation with grippers, using a dexterous hand introduces specific difficulties in tasks such as cable grasping, sliding, in-hand bending, etc, for which no dedicated task definitions, benchmarks, or success metrics exist.

Our initial exploration was based on using a multi-fingered hand-in simulation to perform some basic cable manipulations.

Due to the difficulty of high-dimension control and limited hand motion data, we used Reinforcement Learning to train an agent based on an anthropomorphic hand with 20 degrees of freedom. We proposed five tasks, including in-hand cable sliding from left to right and from right to left, object lifting, cable end-tip position control, and cable bending.

We achieved relatively good results with over 60% success rate on these tasks.

However, implementing the system in the real world becomes an entirely different case, and three things are needed: 1. a benchmark for dexterous cable manipulation, 2. a better-designed multi-fingered hand for dexterous cable manipulation, and 3. a controller that can perform high-dimensional manipulation in the real world.

We provided three solutions:

(1) We first defined and concluded a series of dexterous cable manipulation tasks into a taxonomy covering most one-hand cable manipulation short-horizon primitives and long-horizon tasks. This proposed taxonomy revealed that thumb-index composition is critical for cable manipulation and decomposed long-horizon tasks into shorter primitives.

(2) We designed a new five-fingered hand with 25 degrees of freedom. It has two symmetric thumb-index compositions and a rotatable joint on each fingertip, which allows it to perform tasks that are even difficult for humans.

Besides, we created a demonstration data collection pipeline for this hand.

(3) We defined a finite state machine based on collected demonstrations of short-term primitives.

The hand can robustly replay 7 short-horizon primitives with over 90% success rate on cables of the same material and over 72% on cables of different materials. For long-horizon tasks which need combinations of at least three primitives, the hand can replay four long-horizon tasks with over 75% on cables of the same material.