Pose and shear-based tactile servoing

John Lloyd and Nathan F. Lepora
School of Engineering Mathematics and Technology,
University of Bristol, UK
Bristol Robotics Laboratory, Bristol, UK

Abstract: Tactile servoing is an important technique because it enables robots to manipulate objects with precision and accuracy while adapting to changes in their environments in real-time. One approach for tactile servo control with high-resolution soft tactile sensors is to estimate the contact pose relative to an object surface using a convolutional neural network (CNN) for use as a feedback signal.

In this paper, we investigate how the surface pose estimation model can be extended to include shear, and utilize these combined pose-and-shear models to develop a tactile robotic system that can be programmed for diverse non-prehensile manipulation tasks.

In future, pose and shear-based tactile servoing could enable many object manipulation tasks and the fully-dexterous utilization of multi-fingered tactile robot hands.

1) Gaussian density neural network model that predicts contact SE(3) pose and post-contact shear with uncertainty

2) Discriminative Bayesian filter that reduces the error and uncertainty of the SE(3) pose-and-shear predictions

3) Feedforward-feedback control methods using velocity control driven by tactile pose-and-shear estimation for tactile servo control, supplemented with controllers for goal-based tasks such as object pushing.

4) Application of these techniques to single- and dual-arms for object tracking, surface following, single-arm pushing and dual-arm pushing.

IJRR paper

arXiv preprint

Demonstrations

Perception algorithm

CNN with regression and GDN architectures used for surface contact pose estimation. (a) Convolutional base for CNN and GDN models. (b) Convolutional block sub-layer structure. (c) CNN multi-output regression head. (d) GDN PDF estimation head.

Control algorithm

(a) Tactile servoing controller used for all tasks, which is the sole controller for the object tracking and surface following tasks. (b) For the tactile pushing controller the tactile servoing controller is supplemented with a target alignment controller.

Open-source robotics

Code repository at github.com/dexterousrobot

github.com/dexterousrobot/tactile_learning/tree/mdn

bitbucket.org/nlepora/tactip-hardware

github.com/dexterousrobot/tactile_servo_control

github.com/dexterousrobot/tactile_data_shear

Citation

@article{lloyd2023pose,
  title={Pose and shear-based tactile servoing},
  author={Lloyd, John and Lepora, Nathan F.},
  journal={The International Journal of Robotics Research}, 
  year={2023},
  doi={}
}