Teleoperating a dexterous robot should feel like moving your own hands. In practice, it often feels like driving a car while wearing oven mitts.

VR headsets like Meta Quest 3 can track 26 hand joints in real time — no external mocap, no mechanical exoskeleton. But vision-only teleoperation has a fatal blind spot: the operator cannot feel contact. When a 22-DoF hand grasps a slippery ball or inserts a peg with sub-centimeter clearance, small tracking errors become crushing forces, slip, or dropped parts. The embodiment gap between human hands and robotic topology makes kinematic mapping alone insufficient for contact-rich tasks.

In DexTeleop-0: Force-Aware Bimanual Dexterous Teleoperation with Ego-Centric Perception towards Shared Autonomy — from Nanyang Technological University — researchers propose a tactile-driven shared autonomy layer that sits on top of standard VR tracking. Coarse human intent stays in charge; a real-time optimization loop adds small, force-compliant corrections from fingertip tactile sensing. Evaluated on a 56-DoF bimanual platform (dual UR7e arms + 22-DoF Sharpa Wave hands) with an Isaac Sim digital twin, DexTeleop-0 raises final-stage success from 4% to 97% on simulated ball assembly and achieves 74–77% completion on long-horizon real-world bimanual tasks — while cutting contact forces by more than half.

The Problem: Vision Tracks Intent; Touch Confirms Reality

High-quality dexterous demonstration data is the fuel for imitation learning, VLA pretraining, and industrial deployment. Teleoperation remains the primary collection channel — but three barriers persist:

1. Embodiment gap

   Human hands and 22-DoF robot hands differ in link lengths, joint ordering, and workspace. Pure kinematic retargeting drifts during pinch, bimanual overlap, and in-hand rotation.

2. No force transparency

   Leader–follower rigs (ALOHA, GELLO) give haptics for parallel grippers but don't scale to multi-fingered hands. VR tracking is flexible but tactile-blind — operators cannot perceive slip, excessive pressure, or asymmetric multi-finger loading.

3. Low data efficiency on contact-rich tasks

   Without closed-loop force regulation, operators retry failed grasps endlessly. Destructive contact (31+ N on a ball grasp) produces unusable demonstrations and risks hardware damage.

Prior tactile-enhanced policies (FTP-1, SaTA) treat touch as a policy input. DexTeleop-0 treats touch as a teleoperation corrector — adjusting the operator's command before it reaches the robot.

The Solution: Track, Retarget, Then Balance Forces

DexTeleop-0 is a three-stage pipeline, illustrated on the project page:

The core insight (Fig. 1 in the paper):

Before balancing, misaligned finger contacts generate asymmetric tactile forces — the object slips or ejects. After balancing, real-time tactile feedback reshapes hand posture into a force-stable configuration while preserving the operator's overall intent.

Unified QP objective combines:

• Localized fingertip force regulation (keep each contact within [f_min, f_max])
• Multi-contact force-torque balance (prevent net torque that induces rotation/slip)
• Joint limit and slew-rate constraints

The operator's trajectory is the nominal path; physics projects it onto a force-compliant manifold.

Putting It to the Test: Isaac Sim + Real Sharpa Wave Hardware

Platform (matched sim-to-real)

Component	Specification
Arms	2× Universal Robots UR7e (6 DoF each)
Hands	2× Sharpa Wave (22 DoF each)
Total controlled DoF	56
Perception	Meta Quest 3 egocentric tracking
Simulation	NVIDIA Isaac Sim 4.5 digital twin
Force balance loop	30 Hz QP solver

Evaluation tasks

Domain	Task	Challenge
Sim	Ball Assembly	Dynamic grasp, slippery sphere, 3 stages
Sim	Stir in Cup	Bimanual tool use, delicate contact
Real	Gear Mesh	Sub-centimeter alignment
Real	Peg Insertion	Tight clearance insertion
Real	Food Sorting	Irregular objects, 4-stage bimanual
Real	Tube Operation	Long-horizon fluid transfer

Operators: 5 inexperienced + 2 professional, 5 trials per task per method.

Baselines: No Residual (pure kinematic mapping), Joint-Level PD (local force damping), Force Tracking only (no global balance), DexTeleop-0 (full method).

The Results: Where Contact Is Hardest, Gains Are Largest

Simulation — Ball Assembly

Method	Stage 3 Success	Mean Tactile Force
No Residual	4%	31.58 ± 10.48 N
Joint PD	1%	29.96 ± 9.75 N
Force Tracking only	78%	11.03 ± 5.01 N
DexTeleop-0	97%	11.15 ± 5.01 N

Without residual control, rigid kinematic overrides blast the ball out of the grasp at extreme pressures. DexTeleop-0 maintains stable multi-finger contact through all three stages.

Simulation — Stir in Cup

No Residual achieves 60% final success but at 15.56 N mean force — roughly 2× DexTeleop-0's 7.93 N. High force risks cup damage; DexTeleop-0 prioritizes force-safe compliance at comparable success (59%).

Real robot — Gear Mesh

Method	Stage 2 (assembly)
No Residual	11.43%
Joint PD	37.14%
DexTeleop-0	57.14%

Real robot — Food Sorting (4 stages)

Method	Stage 4	Force
No Residual	48.57%	6.92 N
Force Tracking	57.14%	5.21 N
DexTeleop-0	74.29%	4.94 N (lowest)

Real robot — Tube Operation (long-horizon bimanual)

DexTeleop-0: 100% Stage 2, 77.14% Stage 3 vs No Residual's 34.29% — global force-torque balance prevents multi-contact distribution mismatch during fluid handling.

Why This Matters for Sharpa Wave

DexTeleop-0 is built on Sharpa hardware as the evaluation platform — not as an afterthought.

Native 22-DoF retargeting target Hand retargeting maps 26 VR-tracked joints to Sharpa Wave's 22-DoF vector space using DexPilot-inspired optimization — the same embodiment standard as EgoScale, Do as I Do, and CAIP. Teleoperation data collected on Wave is directly compatible with downstream training pipelines.

Tactile sensing as a first-class control input Wave's Dynamic Tactile Array (>1,000 pixels per fingertip, 6-DoF force/wrench API) feeds the 30 Hz QP loop. This is the collection-time counterpart to FTP-1's pretrained tactile policies and SaTA's spatially grounded touch — touch matters at every stage of the stack.

Complements Sharpa's DexEMG research DexEMG removes the VR headset via wearable EMG; DexTeleop-0 keeps accessible Quest 3 tracking but closes the force loop both approaches need. Together they sketch a teleoperation portfolio: portable EMG for experts, VR + tactile shared autonomy for scalable demo collection.

Isaac Sim alignment Matched digital twin in Isaac Sim 4.5 enables trajectory replay under randomized mass/friction before hardware deployment — consistent with Sharpa's NVIDIA Isaac GR00T integration and Tacmap tactile sim-to-real work.

High-quality data for CraftNet and beyond

The paper's conclusion explicitly targets using DexTeleop-0 trajectories to train generalizable imitation policies. This is the teleoperation front-end for CraftNet's "Midas Touch for Data" — converting human intent into force-aware, robot-complete demonstrations without destructive contact.

Key design lesson from the authors:

Incorporating localized tactile corrections and physical force-balancing directly into the tracking optimization loop is more critical for closing the embodiment gap than merely increasing baseline tracking resolution.

The Takeaway: Teleoperation That Respects Physics

Dexterous data collection does not need expensive mocap rigs or kinesthetic leader arms. It needs accessible tracking (Quest 3) plus physical awareness (Wave tactile + QP force balance).

For teams building on Sharpa Wave: DexTeleop-0 shows how to turn VR teleoperation from a fragile kinematic mirror into a force-compliant demonstration factory — the upstream data pipe that Do as I Do fills from internet video and EgoScale scales from human pretraining.

Track with your eyes. Correct with your fingertips. Train on what actually worked.

Project page: henryhcliu.github.io/dexteleop-0