POISE translates and rotates objects toward commanded palm-relative 6D poses on the Sharpa Wave hand, reaching one target after another without resets.
Many manipulation tasks begin once the object is already in hand: shifting the grip along a tool, exposing the face to be inserted, or lining a tool's axis up with the workpiece. Each requires changing the object's position and orientation together without letting go. Most learned in-hand skills, however, cover only reorientation, rotation about an axis, or constraied translation.
Researchers from Sharpa, the University of Sydney, and the University of Hong Kong have introduced POISE (Palm-relative Object reaching In SE(3)), a sim-to-real reinforcement learning framework for this problem. A dexterous hand moves a grasped object to any palm-relative 6D pose using finger motions alone. It then moves on to the next target without releasing the object or resetting.
Core Idea: In-Hand 6D Reaching as a Primitive
The wrist is fixed, the object is already grasped, and the target is a 6D pose in the palm frame. Coupling translation with rotation forces contact switching and finger gaiting while the object stays held. Every pose the hand reaches must also leave a grasp that can make the next move.
A PPO policy trained in Isaac Lab takes the estimated object pose, the target pose, proprioception, and an object-shape descriptor, and outputs finger joint commands. On hardware, FoundationPose estimates the object pose from a single RGB-D camera, and nothing else changes.
Overview of POISE. Diverse stable grasps initialize simulation training; FoundationPose supplies object-pose feedback on hardware.
Technical Highlights
Diverse stable-grasp resets. Training episodes start from a cache of physics-validated grasps spread across the in-hand workspace. The policy therefore learns from many grasps instead of exploring outward from a single one.
Geometry, not object IDs. A single policy is conditioned on a compact shape descriptor and trained across several shapes and sizes.
Adaptive 6D curriculum. Target magnitudes grow from 5° / 10 mm to 180° / 30 mm as the policy succeeds.
Grasp-preserving reward. Alongside pose error, the reward asks the grasp to keep resisting forces and torques in every direction, so the object is still well held after each target.
Experimental Results
In simulation, each design choice was tested in isolation:
-
Diverse resets: success from held-out grasps rises from 40.1% to 51.5%. Recovery after a drop rises from 33.8% to 72.9%.
-
Curriculum: success on full-range targets rises from 6.2% to 59.5%.
After a drop, the policy trained with diverse resets (top) rebuilds its grasp and reaches the target. The policy trained from a single grasp (bottom) stalls.
On a 22-DoF Sharpa Wave hand:
-
Successive reaching across geometries. On a cube, a hexagonal prism, a square bifrustum, and a hammer, every rollout reached at least five targets in a row without reset. Hammer targets involved changes of up to 93 mm and 180°.
-
Changing gravity. The same policy kept reaching under three wrist orientations, with average errors of 5.3–7.4 mm and 3.2°–4.4°.
-
Disturbance recovery. When the object was pushed off its target, the policy reorganized its contacts and returned the object to the target.
-
Grasp reward. It raised success on three-target sequences from 2/10 to 8/10.
Continuous random reaching, disturbance recovery, and reaching with the wrist turned sideways.
Two Core Advantages Sharpa Wave Brings to This Study
-
Faithful control for sim-to-real. The policy is trained only in simulation and drives the real hand through the same joint-command interface. It reaches targets within millimeters across wrist orientations.
-
An anthropomorphic 22-DoF hand. Reaching a 6D pose inside the hand comes down to switching contacts. With five fingers and an opposable thumb, some fingers can hold the object while others reposition. This holds even on a 215 mm hammer, and even when the palm no longer supports the object.
Conclusion
POISE shows that policies trained entirely in simulation can move a held object to arbitrary palm-relative 6D poses on a real dexterous hand. They do so target after target, across object shapes and wrist orientations. This makes palm-relative SE(3) reaching a practical building block for tool use and general-purpose dexterous manipulation.
Three Key Takeaways
-
The object's pose in the palm is a clean interface between the task and the fingers.
-
Cover the start states, not just the goals. Diverse grasp resets and a goal curriculum make full 6D reaching learnable.
-
A reached pose is only useful if the grasp still holds. Preserving grasp quality took real-world sequence success from 20% to 80%.
