Capturing Motion for Dexterous Manipulation

In previous posts we’ve introduced two new hardware products: The Xicap wrist tracker that combines Vision Pro spatial accessory tracking, ART high precision outside-in tracking and electromagnetic tracking options, and the Xitac sensor that adds tactile perception to motion capture gloves and robotic grippers.

The purpose of these devices is to create state-of-the-art sensors for robot model training. Robotics models are primarily trained from human demonstration data by either capturing human motion while executing a task or by directly guiding a robot using teleoperation. A lot of human motion capture devices exist, but few combine the most important characteristics to accurately capture dexterous manipulation tasks, especially in diverse environemnts like a factory or a home.

Besides working on finalizing the electronics and firmware for these two hardware devices, we are also exploring how to integrate them with an efficient application framework. While on the sensing side itself we integrate the devices with our FusionHub realtime data processing solution, we are researching how to visually best interact with users during training episodes.

For this purpose we created an application for the Vision Pro mixed reality headset that visualizes the state of our mocap system. At the same time this application can also overlay the estimated actions of a robot arm responding in realtime to the motions of the trainer. This is to help the trainer perform motions in a way that can be best used by the robotics model during the later training process. This embodiment is important to allow the model to correspond human motion with robotics arm movements.

Working towards a fully integrated data capture pipeline for robot training, seamless switching between mixed reality, simulation and teleoperation is needed. Using our FusionHub backend this was quite straightforward to implement. The video above shows how we integrate mixed reality tracking and simulation in one application surface. At the moment the system doesn’t yet include teleoperation of a real robot. This will follow soon!

Development of the Xitac Tactile Sensor

Xikaku’s mission is to record skilled human work with precise hand, finger and tactile sensing and use it to train robots for specific tasks. In the diagram below we show the general principle of our tacking solution based on a wrist tracker, tactile finger sensors and visual pose recognition using a headset.

An important part of data collection for humanoid robots is accurately recording hand movements. This is essential for most manipulation tasks that require a high amount of dexterity. Most productive human activities focus rather on deterous and agile upper body and hand motions than movements of the lower body.

A central component of our robotic training solution are the sensors we attach to the fingers of a user. These units are a combination of an IMU - they contain a gyroscope and accelerometer - and a touch sensor. In this article we’ll focus on this tactile sensor.

As shown in the image above, we place a layer of magnetotactic gel over an array of 3D hall sensors. Actuating force on the gel deforms the gel and changes the shape of its magnetic flux. The magnetometers measure this distortion of the magnetic field to obtain data on the pressure and shear force acting on the gel.

At a recent AI Tinkerers demo event in LA, we presented a first prototype of the Xitact tactile sensor. See a video of the excellent presentation by my colleagues Lauren Thomann and Justin Li below.

While creating a full glove using the Xitac sensor is still work in progress, we’ve been making important steps towards building the first prototypes of the device and overcome a few unexpected issues. Sourcing and manufacturing the magnetotactic gel turned out to be a bigger challenge than we had initially assumed.

Our current prototype doesn’t yet use a custom made sensor PCB, instead we’re reusing the base board of our LPMS-B2 Bluetooth sensor. The video below shows first results from operating the sensor in this way. This is a very qualitative result, as I press my finger down on the gel you can see the magnetic field graph changing. This confirms that we’re getting a signal from the magnetic gel, but it doesn’t give us the quantitative information we require. For that we need to switch to our custom made PCB that allows us to look at the magnetic flux differential.

Next step will be to attach such a tactile tracker to each finger, and then fuse together optical tracking data from a headset and inertial data from the sensor. This information will then be paired with the tactile information from each tip and sent to our data acquisition and analysis software FusionHub.

Collaboration with Knoxlabs

We’re happy to announce our first official collaboration here in Los Angeles. Knoxlabs is a leading distributor and integration partner of robotics solutions. Taron, Robert and their team have years of expertise in sourcing the latest and greatest robotics equipment from a wide range of suppliers. On top of their sourcing experience they are also an they also help integrate components into a working system, backed by hands-on engineering experience, not only by finding the best components, but also by providing the knowledge how to make them work together in the best way.

Xikaku is building hardware and software products to revolutionize robotic training and teleoperation. Starting with our deep knowledge of inertial measurement units and mixed reality headset tracking, we’re now applying our skills to provide motion capture solutions to gather the best possible data for training language-action models. With the humanoid robotics industry in the US rapidly evolving we’re looking to provide the tools to finally bring humanoid robots into the places where they’re needed most. Who could be a better partner to embark on this journey together than Knoxlabs!

As a first step we’ll soon launch the Xicap tracker. The Xicap tracker is a wrist worn multi-purpose tracker that can either be tracked visually as an Apple Vision Pro accessory or via electromagnetic tracking. This device will be offered by Knoxlabs as premium distribution and integration partner.

An extension to Xicap tracker will be Xicap glove, a touch sensitive motion capture glove that focuses on the tactile force measurements while its wearer manipulates objects. Stay tuned for updates!