Dexterous reorientation is not simply a sequence of joint targets. A hand must maintain controllable contact while the object rolls, slips, changes support, or moves across fingertips. Policies trained directly in a 22-dimensional joint-action space often spend most of their exploration breaking contact rather than discovering useful finger gaits.
Researchers from Sharpa Robotics introduced TacBPM, a tactile-conditioned behavior prior model that compresses contact-rich manipulation into a reusable latent control interface. Instead of forcing every downstream policy to rediscover rolling, regrasping, and grasp stabilization from scratch, TacBPM distills these behaviors from simple specialist tasks and lets new policies adapt them through compact residual latent actions.
TacBPM spans the full transfer chain: multi-scale specialist training, tactile-conditioned latent distillation, residual-latent reuse on new geometries, commanded-axis rotation, real-hand deployment, and arm–hand Grasp-to-AnyPose transfer.
Core Idea: Separate Reusable Contact Behavior from Task Intent
A sphere is geometrically simple, but changing its diameter changes the hand aperture, fingertip placement, contact-force distribution, and rolling or regrasping regime. TacBPM exploits this controlled variation to learn a broad substrate of hand–object interaction before asking the policy to handle harder shapes and tasks.
The key design is a tactile-proprioceptive behavior prior. Recent joint history, previous targets, tactile contact magnitudes, and 3D contact locations determine which latent behavior is appropriate for the current hand–object state. The downstream policy does not output raw joint commands. It predicts a residual around the prior mean, keeping exploration close to contact-stable behavior while preserving enough freedom for task-specific adaptation.
TacBPM pipeline: multi-scale specialists are distilled into a tactile-conditioned latent controller; downstream policies reuse the frozen prior through residual latent actions; the same paradigm supports in-hand and arm–hand tasks with successful sim-to-real transfer.
Stage 1: Cover the Hand’s Contact Regimes with Simple Specialists
For in-hand reorientation, TacBPM trains eight sphere specialists with scale factors from 0.3 to 1.0, corresponding to grasp diameters of roughly 2.4 to 8 centimeters. This range nearly spans the reachable aperture of the hand. Each specialist learns arbitrary-pose reorientation at its assigned scale, producing distinct fingertip placements, force patterns, and finger gaits without mixing geometry complexity into the teacher stage.
Online multi-teacher distillation then trains one student on states induced by its own rollouts. Each environment is assigned a sphere-scale teacher, and the corresponding expert action becomes the target at the student’s current state. This avoids the distribution mismatch of a fixed offline action dataset and preserves the behavior diversity of the specialist family.
Eight sphere scales expose the hand to a controlled sweep of apertures and contact regimes. The teachers remain geometrically simple so the distilled latent space can focus on reusable rolling, regrasping, and stabilization behavior.
Stage 2: Distill Touch into a Latent Control Interface
The distilled controller contains an encoder, a learned Gaussian prior, and a decoder. During distillation, the encoder sees task information and predicts a posterior latent, while the prior learns to infer useful latent behavior from tactile–proprioceptive history without access to the downstream goal. The decoder maps the latent and hand state to joint targets.
This architecture makes contact part of the action interface. The same nominal task can produce different latent codes when the grasp is stable, fingertip pressure weakens, or contact begins to drift. TacBPM therefore does not memorize one trajectory per object. It learns a distribution of behaviors conditioned on the physical interaction currently unfolding.
Stage 3: Reuse the Prior with Residual Latent Actions
For a downstream task, the tactile prior and state normalization remain frozen. A new policy receives task observations and predicts a residual latent correction around the prior mean. In the in-hand experiments, the decoder is finetuned so the same latent interface can realize useful behavior on anisotropic and irregular geometries without rewriting the contact-conditioned prior.
This division of labor is the central contribution. The prior anchors low-level exploration near viable contact patterns, the task policy specifies what the object should do, and decoder adaptation accounts for how a new geometry turns latent intent into joint motion. The policy can therefore focus on target pose or rotation axis instead of repeatedly relearning how not to drop the object.
Technical Highlights: Why the Prior Transfers
-
Tactile conditioning: Contact magnitudes and 3D contact locations let the prior react when support shifts between fingers or the object begins to slip.
-
Structured exploration: Residual latent actions replace unconstrained exploration in the raw 22-DoF hand-action space.
-
Simple-to-complex distillation: Multi-scale spheres cover contact regimes while leaving harder geometry to downstream adaptation.
-
Frozen prior, adaptable decoder: The behavior distribution remains stable while the latent-to-action mapping adapts to new object shapes.
-
Embodiment-level extension: The same teacher-distillation and residual-latent paradigm is instantiated for a coupled arm–hand Grasp-to-AnyPose task.
Experimental Results: From New Shapes to Six-Axis Commands
TacBPM is evaluated across three increasingly broad settings: In-Hand-to-AnyPose on anisotropic objects, Axis-Conditioned Rotation with six signed Cartesian commands, and Grasp-to-AnyPose with a robot arm and dexterous hand.
-
Generalist prior coverage: The distilled tactile prior reaches 95.62% average success on the seen sphere scales.
-
Contact-regime extrapolation: On held-out scales and unseen shapes, tactile conditioning improves average success from 40.05% to 60.38% over the no-tactile prior.
-
Complex-object transfer: A shared downstream policy reaches 70.00% success across five object geometries, while matched raw-action PPO remains near failure at 1.00% in the multi-object setting.
-
Axis-conditioned reuse: One policy follows +x, −x, +y, −y, +z, and −z commands while preserving contact, then transfers to the physical hand without real-robot policy finetuning.
-
Arm–hand generalization: Across four held-out tool geometries, TacBPM averages 96.73% success in simulation, compared with 49.24% for raw-action learning.
Real-robot commanded-axis rotation on a multiface object: the controller must change rotation direction while maintaining usable contacts on faceted geometry. Tactile feedback helps the prior distinguish stable rolling from drift, stuck grasps, and impending contact loss.
Beyond In-Hand Rotation: Grasp, Lift, Transport, and Reorient
Grasp-to-AnyPose expands the problem from an object already supported in the hand to a complete arm–hand sequence. The robot must acquire a tool, lift it, transport it, and reach a sampled 6D goal while preserving useful fingertip contact. TacBPM instantiates the same prior-centered paradigm with scale-randomized rubber-hammer teachers, then evaluates transfer to a small hammer, blue brush, marker, and mallet hammer.
Initial–Grasp–Lift–Goal sequences across the rubber-hammer teacher family, held-out simulated tools, and a real-robot rollout. The task couples arm motion, gravity loading, tool inertia, and hand reorientation.
SharpaNorth arm–hand transfer: the robot grasps an asymmetric hammer, lifts it from the table, and adjusts its pose after support is removed. Small contact errors can compound through the long-horizon sequence, making grasp preservation part of the control problem rather than a one-time initialization.
The quantitative arm–hand comparison is performed in simulation on novel tools. The hardware demonstrations provide qualitative evidence of successful sim-to-real transfer for the complete grasp–lift–goal sequence; they are not presented as a large-scale real-robot benchmark.
Two Advantages Sharpa Hardware Brings to This Study
1. Touch is available where the prior needs it. On the Sharpa Wave hand, tactile magnitudes and contact locations provide direct evidence of support changes at the fingertips. In-hand deployment runs proprioceptive and tactile feedback at 20 Hz and does not use object-pose feedback in the action loop, forcing the policy to maintain contact from hand-local sensing rather than external object tracking.
2. The platform exposes both hand-level and arm–hand transfer. The 22-DoF anthropomorphic hand supports dense multi-finger contact, rolling, and regrasping across irregular objects. SharpaNorth then extends the same research question to coupled grasp acquisition, lifting, transport, and goal-pose reaching, where tool inertia and wrist motion continually reshape the contact state.
Conclusion
TacBPM shows that tactile feedback can do more than improve a task-specific policy. It can organize a reusable behavior prior that downstream policies treat as a compact action space. Distilling stable contact behavior first changes the exploration problem: new policies search over task-relevant latent corrections instead of rediscovering low-level finger coordination from raw joint commands.
Three key takeaways:
-
Contact behavior can be reused. Rolling, regrasping, and stabilization need not be relearned independently for every object and task.
-
Touch belongs inside the behavior prior. A proprioceptive motion manifold alone cannot tell whether the object is supported, drifting, or approaching contact loss.
-
Residual latent control separates skill from intent. The prior preserves viable interaction patterns, while downstream policies specify target pose, rotation axis, or arm–hand goal.
Explore the TacBPM project page and demos and read the preprint.
