Smart e-skins and machine learning for soft robot perception

Abstract

Perception is the foundation for intelligent robots to effectively explore environments and interact with users. Bio-inspired soft robots can exploit the compliance of their bodies, consequently demonstrating advantages in terms of safety during humanrobot interaction and operability in unstructured scenarios. However, this very feature dramatically increases the complexity of perception, which prevents soft robots from wide adoption in practical applications. This thesis aims at this challenge to implement a comprehensive study in coupling field simulation, sensor design and fabrication and learning-based perception algorithms. The major contributions of this thesis can be summarised as follows.

First, coupling field simulation (CFS) was developed to integrate sensory systems into mechanical structures, which is critical for developing perceptive soft robots but neglected by most existing studies. The proposed simulation method was demonstrated through a 16-electrode capacitive sensor array deployed on a soft robot manipulator. The understanding of the sensor behaviours was built by simulating the sensor responses to a range of deformations using CFS. One of the most important applications of CFS is to generate annotated samples which can be used to train learning-based perception algorithms. The trained learning models could be transferred to practical scenarios without the need for tremendous annotated real-world data, thus reducing time and costs for data acquisition. Two case studies for applied force estimation and deformation classification were performed with annotated data generated by CFS to demonstrate its potential in developing learning-based perception methods.

On soft robot proprioception, previous studies only achieved low proprioceptive geometry resolution (PGR), thus suffering from loss of geometric details (e.g., local deformation and surface information) and limited applicability (e.g., only applicable to prescribed simple deformations). This thesis proposed a high PGR soft robot proprioception system, which encapsulates an intrinsically stretchable capacitive e-skin (SCAS) and a capacitance-to-deformation transformer (C2DT), to endow full-geometry, millimetrelevel bodily awareness to soft robots. The SCAS has a redundant planar electrode layout that forms a sequence of capacitors sensitive to deformation across the entire soft robot. The C2DT based on transformer architecture can explore the dependency over the SCAS signals and recover deformation from the signals. The proposed proprioception framework synergistically combining the SCAS and the C2DT can achieve real-time (30 fps), high PGR (3,900) full-geometry deformation reconstruction with high accuracy (2.322 ± 0.687 mm CD error) under a range of complex deformations on a 20 × 20 × 200 mm soft manipulator. This high PGR was not demonstrated previously and is one or two orders of magnitude improvement over previous methods.

Regarding tactile sensing (one of the most important exteroceptions), previous studies ignored the impact of deformation on the sensing data, making them a mismatch to practical scenarios where the geometric and tactile variations are coupled in the sensor signals. The thesis performed a preliminary exploration of tactile sensing with severe interference, which is inevitable in some practical applications. A simplified SCAS was proposed to achieve touch recognition with the interference induced by deformation while reducing the complexity of fabrication, deployment and wiring. The proposed method was validated on a pneumatic robotic platform. Contact location estimation was successfully achieved at low spatial resolution in various inflation conditions with 99.88% of classification accuracy. Moreover, deformation sensing with the interference induced by physical contact was also demonstrated by the coordinates estimation of 5 visual markers deployed on the soft robot platform. The C2DT was employed to perform marker tracking with the information of the first frame in a trajectory as prior knowledge and achieved 2.905±2.207 mm AD error, which shows the potential of the simplified SCAS to simultaneously detect internal and external stimuli.

In summary, this thesis presented a framework for soft robot perception, which comprises stretchable capacitive e-skins to translate geometric and tactile variations to capacitance data and neural architectures to interpret the capacitance data to desired parameters. High PGR morphological reconstruction and tactile sensing with interference were demonstrated, paving the way towards the autonomy of soft robots.