From Fixed Finger Gaits to Closed-Loop Physical Reasoning: WM-Craftnet Brings Perturbation-Robust Dexterity to a Human-Sized 22-DoF Hand

In dexterous robotics, rotating and adjusting objects inside the hand remains one of the hardest contact-rich skills. Policies trained on narrow object sets or fixed initial poses often memorize object-specific finger gaits. When pose shifts, force disturbances, or unseen geometries appear, those controllers quickly fail.

Recently, researchers from Sharpa Robotics introduced WM-Craftnet, a world-model-conditioned framework for robust and generalizable in-hand manipulation. Rather than treating vision and touch as static observations, WM-Craftnet learns a World Synesthesia Model (WSM)—an action-conditioned recurrent state that fuses wrist depth, tactile contact, proprioception, and control history into a reusable physical prior for closed-loop dexterity.

Core Idea: A Deployable Visuotactile State Estimator

The central insight behind WM-Craftnet is that robust in-hand manipulation requires more than fusing raw sensors. The policy must infer object geometry, contact evolution, drift, and slip under partial and noisy observations.

WM-Craftnet trains a Dreamer-style recurrent state-space model from proprioception, noisy depth, tactile signals, actions, and rewards. Importantly, the world model is not used for latent imagination or policy optimization. Instead, its deterministic recurrent feature serves as deployable task context for a PPO actor—turning depth, touch, and action history into a denoised, temporally coherent interaction state.

During training, noisy augmented depth is fed into WSM while clean wrist-depth targets supervise reconstruction. This design encourages the model to recover task-relevant hand-object geometry from realistic sensor corruption—critical for real-robot deployment where wrist depth is often incomplete and noisy.

WM-Craftnet pipeline

WSM reconstructs clean hand-object geometry from noisy wrist-depth observations, providing a deployable visuotactile state for real-robot control.

Technical Highlights: Why WSM Works

1. Clean geometric supervision: WSM reconstructs clean depth from noisy inputs, providing a denoised geometric state rather than relying on raw corrupted depth.
2. Action-conditioned recurrent memory: The performance gain comes from predictive supervision plus temporal memory—not simply adding depth or tactile inputs to the policy.
3. Object-ID-free generalization: The recurrent state organizes interaction regimes by geometry and contact affordances without object identifiers, enabling one policy to rotate diverse objects.
4. Reusable synesthetic prior: WSM pretrained on nine z-axis objects transfers to 49 new objects, dramatically improving downstream learning efficiency and stability.

Training reward curves (z-axis, WSM ablation, x/y-axis)

WSM recurrent state t-SNE

WSM ablation training curves

Experimental Results: A Win for the Generalist

WM-Craftnet was evaluated on a human-sized 22-DoF five-finger Sharpa Wave Hand across simulation and real hardware, covering z-, y-, and x-axis rotation, unseen objects, perturbation recovery, and tool-use-style manipulation.

• Simulation breakthrough on multi-object z-axis rotation: Full WSM reaches 753.3 return and 1.293 rotation rate, far above raw-sensor baselines (236.1–386.9 return) and recurrent-history controls (LSTM: 621.0, GRU: 497.8).
• Real-robot dominance: On duck rotation, WM-Craftnet achieves 16.18 rad versus 2.76 rad for In-Hand Rotation with WSM-denoised depth. Across duck, cross block, corner block, and an unseen double-notched block, WM-Craftnet reaches 10/10, 10/10, 10/10, and 8/10 success rates.
• Large-scale generalization: A WSM prior pretrained on nine objects enables downstream control over 49 new objects, reaching 9.37 ± 0.13 rad per episode versus 3.28 rad without the prior, while fall rate drops from 6% to 0.3%.
• Harder rotation modes: On tool-like y-axis objects, WM-Craftnet nearly doubles the best baseline rotation rate (0.545 → 1.025). On contact-sensitive x-axis rotation, it reaches 432.8 return with the lowest fall rate.
• Robustness under stress: WM-Craftnet succeeds in 175/200 real-world rotation trials across 20 objects, recovers from challenging initial poses and in-process force perturbations, and zero-shot rotates unseen geometries.

Two Core Advantages Sharpa Wave Brings to This Study

1. High-Fidelity Visuotactile Hardware for Sim-to-Real Transfer

WM-Craftnet's success depends on transferring a policy trained in simulation directly to a physical hand without object-specific fine-tuning. Sharpa Wave provides the sensing and actuation fidelity needed for this pipeline:

• Integrated wrist depth and tactile feedback give the policy complementary global geometry and local contact evidence under real sensing noise and latency.
• Human-sized 22-DoF anthropomorphic morphology supports dense fingertip contacts, in-hand reorientation, and complex contact reallocation required for x- and y-axis rotation.
• Predictable hardware behavior reduces the sim-to-real gap, allowing WM-Craftnet's learned WSM state to deploy robustly on physical hardware.

2. An Anthropomorphic Platform for Contact-Rich Dexterity

WM-Craftnet highlights adjustment behaviors—re-centering, uprighting, palm-to-finger transfer, and perturbation recovery—before sustained target-axis rotation. An anthropomorphic five-finger hand is essential for realizing these skills:

• Opposable-thumb, human-like kinematics enable coordinated multi-finger adjustment on asymmetric objects such as bulbs, corner blocks, and elongated tools.
• Rich contact workspace supports gravity-invariant compact-grasp rotation and tool-use-style screwdriver translation and fast axial spinning.
• Closed-loop finger control lets the policy adapt online when objects drift, slip, or leave the nominal contact mode—rather than replaying a fixed open-loop gait.

Conclusion

WM-Craftnet shows that by learning an action-conditioned World Synesthesia Model in large-scale simulation, it is possible to induce highly robust dexterous manipulation skills—multi-object rotation, unseen-object transfer, perturbation recovery, and tool-use-style control—on a real human-sized dexterous hand. This opens a scalable path toward robots that can manipulate everyday objects with human-like fluency under real-world sensing noise and contact uncertainty.

Three key takeaways:

1. World models can serve as reusable synesthetic priors. WSM transfers physical interaction knowledge learned from a limited set of objects to many unseen objects, enabling more efficient scale-up without relying on object IDs.

2. Denoising depth is a practical bridge for sim-to-real deployment. By reconstructing clean wrist-depth targets from noisy depth inputs, WSM narrows the real-world vision gap and provides the policy with more reliable hand-object geometry.

3. For dexterous robots, world models are not only for dreaming of the future.WM-Craftnet shows that an equally important role is learning a better representation of the present: fusing vision, touch, proprioception, and action history into a robust policy context for closed-loop manipulation.

Explore more demos and ablations on the project page: https://wmcraftnet.github.io/

Preprint： https://arxiv.org/abs/2609.07002

Authors:Jie Yin, Zeyuan Zhao, Xiaojing Tan, Yang Liu, Chiyu Wang, and Xinyang Gu, Sharpa Robotics.

Accepted to CoRL 2026 (Austin Texas, USA)