View Project

Norwegian AI Directory

GentleMAN-Gentle and Advanced Robotic Manipulation of 3D Compliant Objects


GentleMAN will result in a novel robot control and learning framework enabling real-world manipulation of 3D compliant objects. This framework will be based on visual and force/tactile sensing modalities and multi-modal learning models by careful balance and tighter integration between the components responsible for object localization and pose estimation, based on visual information, and the one responsible for manipulation based on the force/tactile information. The robotic manipulation of 3D compliant objects remains a key, yet relatively poorly-researched, field of study. Currently, most approaches to robotic manipulation focus on rigid objects. These are primarily vision-based and require a 3D model of the object or attempt to build one. The interaction of a robot with 3D compliant objects is one of the greatest challenges facing robotics today, due to complex aspects such as shape deformation during manipulation, and the need for real-time perception of deformation and the compliancy of the objects. To these are added coordination of the visual, force and tactile sensing required to control and accomplish specific manipulation tasks. These challenges become even more acute if the objects are slippery, made of soft tissue, or have irregular 3D shapes. Such objects are common in the agriculture, manufacturing, food, ocean space, health and other sectors in both industrial and non-industrial settings. The GentleMAN addresses these challenges by focusing on providing robots with advanced manipulation skills that reproduce human-like movements and fine motor skills. Robots will thus learn intelligently how to induce and apply the necessary manipulative forces while generating occlusion-resilient vision control, real-time 3D deformation tracking and a shape-servoing strategy. A highly qualified and expert interdisciplinary consortium, consisting of SINTEF, NTNU, NMBU, INRIA, MIT and QUT has been assembled to conduct the proposed research.

Project leader: Ekrem Misimi

Started: 2019

Ends: 2023

Category: Teknisk-industrielle institutter

Sector: Instituttsektor

Budget: 16000000

Institution: SINTEF OCEAN AS

Address: Trondheim