Children learn dexterity by watching. A toddler sees someone whisk eggs, pour water, or hammer a nail — and eventually imitates. Robots, by contrast, learn almost exclusively by doing: expensive teleoperation, simulation rollouts, or carefully staged demonstrations.

The observational data is already there. YouTube, egocentric datasets, and generative video contain trillions of hours of human hand–object interaction. The bottleneck is not availability — it is conversion: turning noisy monocular RGB into robot-complete action trajectories on multi-fingered hands.

In Do as I Do: Dexterous Manipulation Data from Everyday Human Videos — a UC Berkeley work led by Bhawna Paliwal, Haritheja Etukuru, and William Liang with Pieter Abbeel and Jitendra Malik — researchers present an end-to-end pipeline that reconstructs 4D hand–object interactions from in-the-wild video and retargets them onto 22-DoF Sharpa Wave hands. To the authors' knowledge, it is the first complete path from internet video to real dexterous hand rollouts — producing 500 verified trajectories across 20 manipulation verbs and deploying 10 tasks on a bimanual UR3e + dual Sharpa Wave platform at 50 Hz.

The Problem: Watching ≠ Doing (Yet)

Scaling dexterous robot data faces two structural barriers:

1. Hand–object reconstruction from monocular RGB is fragile

2. In-the-wild video has motion blur, occlusion, depth ambiguity, and arbitrary objects. FoundationPose-style trackers lose lock under mild blur; joint reconstruction methods assume lab conditions or closed object taxonomies. Without robust 4D reconstruction, human video stays unusable as training signal.

3. Retargeting breaks on noisy references

4. Prior dynamics-aware retargeting (SPIDER, RL-based trackers) assumes clean MoCap ground truth. Reconstructed internet video introduces temporal discontinuities, misaligned contact, and impossible initial states — causing 75% failure rates on naive sampling-based optimization.

5. Teleoperation does not scale

6. Expert operators, specialized rigs, and per-task collection cannot match the diversity of a single hour of human cooking footage — let alone the internet.

Do as I Do asks: Can we close the watch-to-do gap using only monocular RGB, with no grasp priors and arbitrary rigid objects?

The Solution: Reconstruct, Then Physically Retarget

The pipeline has two stages, visualized on the project page:

Stage 1 — Guided diffusion object tracking

SAM 3D generates per-frame object meshes — but applied independently, meshes drift and lose temporal coherence. Do as I Do fixes shape at an anchor frame and biases pose sampling toward the previous frame via flow-matching guidance, with adaptive pose guidance derived from 2D point-track rotational velocity. On 150 in-the-wild videos, human raters prefer this tracking over FoundationPose 67% of the time (often unanimously).

Stage 2 — Robust retargeting for noisy references

Building on SPIDER's annealed MPPI framework, three innovations handle reconstruction noise:

Together, retargeting success on noisy in-the-wild references rises from 25% → 71%.

Putting It to the Test: 20 Verbs, 500 Trajectories, 10 Real Deployments

Reconstruction benchmarks (SOTA)

Retargeting benchmarks

Data sources for 500 verified trajectories

20 manipulation verbs: placing, picking, scrubbing, spreading, squeezing, ironing, painting, dusting, digging, erasing, pouring, writing, whisking, stirring, poking, tamping, drilling, hammering, cutting, basting.

Real-world deployment (Sharpa Wave)

Ten motions executed on bimanual UR3e arms + dual 22-DoF Sharpa Wave hands — whisking, pouring, dusting, squeezing, tamping, erasing, stirring, hammering, spreading, picking — spanning tripod, power, ventral, and parallel extension grasps.

The Playbook: Why 95% of Internet Video Gets Filtered Out

A practical contribution for teams scaling human data (including EgoScale practitioners): analyzing 2,000 clips from 100DOH (already filtered for hand–object interaction):

Implication: a ~20× penalty if you dump raw internet video into training without preprocessing. Do as I Do's filtering playbook — boundary checks, shot continuity, camera motion, SAM 3D failure modes — is essential infrastructure for any human-video-to-robot pipeline on Sharpa hardware.

Why This Matters for Sharpa Wave

Unlike papers that mention anthropomorphic hands in passing, Do as I Do is built on Sharpa Wave end-to-end.

Native 22-DoF retargeting target

All reconstruction, simulation (MuJoCo Warp @ 200 Hz), and real deployment use the Sharpa Wave — the same 22-DoF, human-scale hand that EgoScale retargets human keypoints into and that CAIP evaluates on Dexmate Vega + dual Wave. The embodiment gap is minimized because the target hand matches the source human morphology.

First internet-to-real pipeline on Sharpa

Prior work (DexMV, DexImit, VideoManip) stops at simulation or requires depth/MoCap. Do as I Do completes the loop: YouTube clip → reconstructed trajectory → physics retarget → real Sharpa Wave rollout. This is the data factory CraftNet's "Midas Touch" vision needs — turning abundant human video into robot-executable demonstrations.

Bimanual dexterity at 50 Hz

Real deployment uses coordinated dual-arm control with Wave's 22-DoF per hand — whisking, stirring, hammering require bimanual coordination impossible on parallel grippers. Wave's >4 Hz gesture speed and >20 N fingertip force provide the mechanical envelope for these contact-rich verbs.

Sim-to-real stack alignment

Retargeting runs in MuJoCo Warp (GPU-parallel); Sharpa ships Isaac Sim URDF/USD assets and collaborates with NVIDIA on tactile sim (Tacmap). Do as I Do's simulation-first retargeting fits naturally into the Sharpa + Isaac training ecosystem.

The Takeaway: Human Video Becomes Robot Data

For decades, "Do as I Do" was an AI aspiration — copy human demonstration onto a robot body. Berkeley's new algorithm makes it literal: paste a URL, get a Sharpa Wave trajectory.

For Sharpa OEMs, researchers, and the Isaac GR00T ecosystem, the implication is direct: the world's largest manipulation dataset already exists. It is filmed on phones. Do as I Do is the converter that turns those pixels into the 22-DoF joint angles Wave was designed to execute.

Watch. Reconstruct. Retarget. Roll out.

Related Sharpa Research:

• EgoScale — Scaling Dexterous Manipulation from Human Video
• CAIP — Action-Centric Vision Pretraining
• FTP-1 — Foundation Tactile Policy
• CraftNet — Hierarchical VTLA