WM-Craftnet: World Synesthesia Model for Robust and Generalizable Dexterous In-Hand Manipulation

2026-09-11

From Fixed Finger Gaits to Closed-Loop Physical Reasoning: WM-Craftnet Brings Perturbation-Robust Dexterity to a Human-Sized 22-DoF Hand

In dexterous robotics, rotating and adjusting objects inside the hand remains one of the hardest contact-rich skills. Policies trained on narrow object sets or fixed initial poses often memorize object-specific finger gaits. When pose shifts, force disturbances, or unseen geometries appear, those controllers quickly fail.

Recently, researchers from Sharpa Robotics introduced WM-Craftnet, a world-model-conditioned framework for robust and generalizable in-hand manipulation. Rather than treating vision and touch as static observations, WM-Craftnet learns a World Synesthesia Model (WSM)—an action-conditioned recurrent state that fuses wrist depth, tactile contact, proprioception, and control history into a reusable physical prior for closed-loop dexterity.

[视频]

Teaser. One policy performs perturbation-robust, object-ID-free in-hand manipulation using wrist depth, tactile feedback, and a World Synesthesia Model recurrent state on the Sharpa Wave hand.

Core Idea: A Deployable Visuotactile State Estimator

The central insight behind WM-Craftnet is that robust in-hand manipulation requires more than fusing raw sensors. The policy must infer object geometry, contact evolution, drift, and slip under partial and noisy observations.

WM-Craftnet trains a Dreamer-style recurrent state-space model from proprioception, noisy depth, tactile signals, actions, and rewards. Importantly, the world model is not used for latent imagination or policy optimization. Instead, its deterministic recurrent feature serves as deployable task context for a PPO actor—turning depth, touch, and action history into a denoised, temporally coherent interaction state.

During training, noisy augmented depth is fed into WSM while clean wrist-depth targets supervise reconstruction. This design encourages the model to recover task-relevant hand-object geometry from realistic sensor corruption—critical for real-robot deployment where wrist depth is often incomplete and noisy.

WM-Craftnet pipeline

[文件]

Real noisy wrist depth WSM predicted depth (real hardware)
[视频] [视频]

Sim: noisy depth input Sim: predicted depth from WSM latent
[视频] [视频]
WSM reconstructs clean hand-object geometry from noisy wrist-depth observations, providing a deployable visuotactile state for real-robot control.

Technical Highlights: Why WSM Works

Clean geometric supervision: WSM reconstructs clean depth from noisy inputs, providing a denoised geometric state rather than relying on raw corrupted depth.
Action-conditioned recurrent memory: The performance gain comes from predictive supervision plus temporal memory—not simply adding depth or tactile inputs to the policy.
Object-ID-free generalization: The recurrent state organizes interaction regimes by geometry and contact affordances without object identifiers, enabling one policy to rotate diverse objects.
Reusable synesthetic prior: WSM pretrained on nine z-axis objects transfers to 49 new objects, dramatically improving downstream learning efficiency and stability.

Training reward curves (z-axis, WSM ablation, x/y-axis)

[图片]

WSM recurrent state t-SNE WSM ablation training curves

[图片] [图片]

Predictive supervision and clean-depth reconstruction shape a reusable interaction state that outperforms raw-sensor baselines and organizes object-dependent contact regimes without object IDs.

Experimental Results: A Win for the Generalist

WM-Craftnet was evaluated on a human-sized 22-DoF five-finger Sharpa Wave Hand across simulation and real hardware, covering z-, y-, and x-axis rotation, unseen objects, perturbation recovery, and tool-use-style manipulation.

Simulation breakthrough on multi-object z-axis rotation: Full WSM reaches 753.3 return and 1.293 rotation rate, far above raw-sensor baselines (236.1–386.9 return) and recurrent-history controls (LSTM: 621.0, GRU: 497.8).
Real-robot dominance: On duck rotation, WM-Craftnet achieves 16.18 rad versus 2.76 rad for In-Hand Rotation with WSM-denoised depth. Across duck, cross block, corner block, and an unseen double-notched block, WM-Craftnet reaches 10/10, 10/10, 10/10, and 8/10 success rates.
Large-scale generalization: A WSM prior pretrained on nine objects enables downstream control over 49 new objects, reaching 9.37 ± 0.13 rad per episode versus 3.28 rad without the prior, while fall rate drops from 6% to 0.3%.
Harder rotation modes: On tool-like y-axis objects, WM-Craftnet nearly doubles the best baseline rotation rate (0.545 → 1.025). On contact-sensitive x-axis rotation, it reaches 432.8 return with the lowest fall rate.
Robustness under stress: WM-Craftnet succeeds in 175/200 real-world rotation trials across 20 objects, recovers from challenging initial poses and in-process force perturbations, and zero-shot rotates unseen geometries.

Continuous multi-object rotation

OOD-to-seen-object recovery

Stable long-horizon rotation (corner block)

Challenging-start recovery (switch)

WM-Craftnet maintains stable z-axis rotation for over one minute on a corner block, and recovers from challenging initial poses through closed-loop visuotactile adjustment.

WSM prior → 49 new objects

Zero-shot unseen geometries

A WSM prior pretrained on nine z-axis objects scales to 49 new objects (9.37 ± 0.13 rad vs. 3.28 rad without prior) and zero-shot rotates held-out shapes without object-specific training.

In-process force perturbation

Real-world perturbation recovery

After external disturbances push objects out of nominal contact modes, WM-Craftnet re-centers them into a controllable workspace and resumes stable target-axis rotation.

Goal-conditioned translation

Fast axial rotation

Beyond benchmark rotation, WM-Craftnet supports tool-use-style manipulation: adjusting a screwdriver toward a target pose while preserving a stable in-hand grasp, and spinning the tool rapidly without slip.

The same policy rotates diverse objects in one uninterrupted real-world run and recovers controllable grasp after out-of-distribution interaction states.

Two Core Advantages Sharpa Wave Brings to This Study

1. High-Fidelity Visuotactile Hardware for Sim-to-Real Transfer

WM-Craftnet's success depends on transferring a policy trained in simulation directly to a physical hand without object-specific fine-tuning. Sharpa Wave provides the sensing and actuation fidelity needed for this pipeline:

Integrated wrist depth and tactile feedback give the policy complementary global geometry and local contact evidence under real sensing noise and latency.
Human-sized 22-DoF anthropomorphic morphology supports dense fingertip contacts, in-hand reorientation, and complex contact reallocation required for x- and y-axis rotation.
Predictable hardware behavior reduces the sim-to-real gap, allowing WM-Craftnet's learned WSM state to deploy robustly on physical hardware.

2. An Anthropomorphic Platform for Contact-Rich Dexterity

WM-Craftnet highlights adjustment behaviors—re-centering, uprighting, palm-to-finger transfer, and perturbation recovery—before sustained target-axis rotation. An anthropomorphic five-finger hand is essential for realizing these skills:

Opposable-thumb, human-like kinematics enable coordinated multi-finger adjustment on asymmetric objects such as bulbs, corner blocks, and elongated tools.
Rich contact workspace supports gravity-invariant compact-grasp rotation and tool-use-style screwdriver translation and fast axial spinning.
Closed-loop finger control lets the policy adapt online when objects drift, slip, or leave the nominal contact mode—rather than replaying a fixed open-loop gait.

暂时无法在飞书文档外展示此内容

Conclusion

WM-Craftnet shows that by learning an action-conditioned World Synesthesia Model in large-scale simulation, it is possible to induce highly robust dexterous manipulation skills—multi-object rotation, unseen-object transfer, perturbation recovery, and tool-use-style control—on a real human-sized dexterous hand. This opens a scalable path toward robots that can manipulate everyday objects with human-like fluency under real-world sensing noise and contact uncertainty.

Explore more demos and ablations on the project page: https://wmcraftnet.github.io/

Preprint：https://arxiv.org/abs/2609.07002

Authors:Jie Yin, Zeyuan Zhao, Xiaojing Tan, Yang Liu, Chiyu Wang, and Xinyang Gu, Sharpa Robotics.

Accepted to CoRL 2026 (Austin Texas, USA)

← Back to list