When you pour water from a bottle, fold laundry, or pull a tissue from a box, your eyes are not doing object classification. They are tracking where hands are going, how contact is evolving, and what must happen next in physical space.

Modern robot policies inherit vision encoders trained for a different job. CLIP and SigLIP learn "this is a mug"; DINOv2 learns geometry and segmentation. These are powerful — but they were never trained to align pixels with end-effector motion. The mismatch propagates into every VLA built on top: π₀.₅, GR00T, OpenVLA all inherit backbones optimized for internet semantics, not contact-rich control.

Robot trajectories would be the natural fix — but at pretraining scale, they barely exist. DROID and Open X-Embodiment are orders of magnitude smaller than human video corpora like Ego4D.

In CAIP: Contrastive Action-Image Pre-training for Visuomotor Control — a Berkeley and NVIDIA collaboration — researchers propose a scalable alternative: extract action labels from 32,000+ hours of egocentric human video using 3D hand poses as a proxy for robot end-effector motion, then pretrain a vision encoder via contrastive action–image alignment. Evaluated on a Dexmate Vega bimanual platform with dual 22-DoF Sharpa Wave hands, CAIP achieves 76% average success across six dexterous tasks — +30 points over the strongest baseline — with only 200 demonstrations per task.

The Problem: Semantic Vision ≠ Manipulation Vision

Visual pretraining for robotics has followed two paths, both incomplete:

Path 1: Internet image–language (CLIP, SigLIP, SigLIP 2) Captures high-level semantics — "soap dispenser," "lamp switch" — but not the hand–object dynamics policies must execute. Saliency maps from SigLIP attend to object identity; they do not focus on grasp affordances or contact regions.

Path 2: Egocentric video without explicit actions (R3M, MVP, VC-1) Trains on human manipulation footage but uses time-contrastive or masked-autoencoding objectives — predicting temporal proximity or missing pixels, not ground-truth motion. The encoder sees hands moving but is never told how they move.

Path 3: Robot-only contrastive learningMethods like Robots Pre-Train Robots align vision with proprioception — but robot data remains scarce.

The gap is precise: downstream visuomotor policies need paired vision and action at pretraining scale. Robot trajectories don't have it. Human video does — if you can recover actions from it.

CAIP's insight: human hand poses are a scalable proxy for robot actions, especially for anthropomorphic end effectors.

The Solution: Contrastive Action–Image Pre-training

CAIP (Contrastive Action-Image Pre-training) aligns three modalities through a SigLIP-style sigmoid contrastive loss.

Action representation (MANO-aligned) Each timestep encodes 42 hand keypoints (21 per hand, including wrists) as SE(3) transforms — the same skeletal structure used in human pose estimation and in EgoScale's retargeting to Sharpa Wave's 22-DoF joint space. Action chunks span T = 64 timesteps (~2 seconds at 30 Hz), expressed as relative SE(3) deltas — analogous to downstream end-effector delta control.

Text-conditioned pooling Language tokens attend over image patches before contrastive alignment, so the encoder learns instruction-specific manipulation features — not generic scene semantics.

Modular backboneCAIP produces a frozen vision encoder transferable to any downstream policy. The paper evaluates with a Qwen3.5-0.8B decoder + flow-matching action head — but the encoder is architecture-agnostic.

The Data: 32,041 Hours of Human Video, 88 Hours of Robot Data

Source	Scale	Role
In-the-wild egocentric video	~31,000 hours	Pose-estimated hand annotations
Lab egocentric (with wrist views)	~1,000 hours	High-quality Manus Metagloves + Vive tracking
Tabletop humanoid manipulation	~88 hours	Embodiment diversity, wrist-view coverage

Total: 32,129 hours of action-labeled pretraining — with robot data comprising <0.3% of the mixture. The diversity of environments and tasks forces the encoder to learn manipulation-centric features that generalize, not memorize a single lab setup.

This is the visual counterpart to EgoScale's action-space pretraining: EgoScale scales motor priors from human video; CAIP scales perception priors from the same abundant source.

Putting It to the Test: Dexmate Vega + Dual Sharpa Wave

Real-world evaluation uses a challenging bimanual setup:

• Arms: Dexmate Vega bimanual manipulator
• Hands: Two 22-DoF Sharpa Wave dexterous hands
• Cameras: ZED X Mini stereo head + two ZED X One S-Wide wrist cameras
• Control: End-effector delta (arms) + absolute joint control (fingers)
• Training data: 200 demonstrations per task (150 for Pour)
• Evaluation: 12 trials per task, six tasks

The paper acknowledges Sharpa and Dexmate for equipment maintenance and software support.

Task	What it tests
Fold Shorts	Deformable manipulation, bimanual coordination
Pour	Liquid handling, controlled tilt
Pick Fruits	Multi-object selection
Dispense Soap	Button press + force control
Turn On Lamp	Fine contact, switch actuation
Pull Tissue	Friction-sensitive extraction

These are everyday dexterity tasks — the kind Sharpa North demonstrated at CES 2026 — not pick-and-place benchmarks solvable with parallel grippers.

The Results: +33 Points Over SigLIP 2, Consistent Across Tasks

Method	Avg. Success
R3M	17.40%
VideoMAE	26.00%
DINOv2	42.00%
SigLIP	42.40%
SigLIP 2	43.40%
MVP	42.00%
CAIP	76.00%

CAIP wins five of six tasks and maintains strong performance everywhere — while baselines swing wildly (MVP: 93.8% on Dispense Soap but 8.3% on Turn On Lamp; DINOv2: 81.3% on Pour but 20.8% on Pull Tissue). Non-manipulation-centric encoders are task-lottery backbones.

Per-task highlights:

Task	SigLIP 2	CAIP
Fold Shorts	4.20%	68.80%
Pour	35.40%	83.30%
Dispense Soap	93.80%	100.00%
Turn On Lamp	50.00%	75.00%
Pull Tissue	25.00%	72.90%

CAIP outperforms the larger SigLIP 2 SO400M despite initializing from the smaller ViT-L backbone — isolating the gain to action-centric pretraining, not model scale.

Representation qualityOn held-out egocentric action classification, CAIP's zero-shot retrieval (no finetuning) beats every baseline's linear probe up to 16 samples per class — the encoder captures transferable action semantics, not dataset-specific shortcuts.

RobustnessUnder lighting perturbations and scene distractors (red book, Hanoi tower), CAIP degrades least in absolute terms — maintaining 43–52% average under perturbation vs 9–28% for baselines.

Why This Matters for Sharpa Wave

CAIP is evaluated on Sharpa hardware for a reason: the action proxy and the robot are designed for each other.

1:1 anthropomorphic action space CAIP's 42-keypoint MANO representation maps naturally to Sharpa Wave's 22-DoF joint space — the same retargeting philosophy EgoScale uses for human-video pretraining. Policies pretrained on CAIP features inherit visual representations already aligned with five-fingered dexterous control, not parallel-jaw semantics.

Bimanual dexterity at human scaleFold Shorts, Pour, and Pull Tissue require coordinated two-hand motion — only possible on platforms pairing dual arms with human-scale 22-DoF hands. Wave's isomorphic kinematics (palm ratio ≈ 0.618) let policies trained on human hand poses transfer without catastrophic embodiment mismatch.

Data efficiency on real hardwareCAIP reaches 76% average with 200 demos per task. For Sharpa OEMs and researchers, this means CAIP-pretrained encoders can slash teleoperation collection — the bottleneck CraftNet and EgoScale also target.

The missing layer in the Sharpa stack

Sharpa's embodied AI spans multiple layers:

Layer	Representative work
Perception pretraining	CAIP (action-centric vision)
Motor pretraining	EgoScale (human video → 22-DoF actions)
Tactile foundation	FTP-1 (cross-sensor touch)
High-frequency control	T-Rex, CraftNet System 0
Spatial touch grounding	SaTA

CAIP fills the vision encoder slot that every VLA needs — producing manipulation-native features instead of repurposed CLIP/DINO weights.

Ecosystem continuity Authors overlap with T-Rex and EgoScale (Dantong Niu, Zhuoyang Liu, Ruijie Zheng, Jing Wang, Danfei Xu, Linxi Fan). The NVIDIA GEAR + Sharpa Wave research line now covers perception (CAIP), motor priors (EgoScale), and tactile reactivity (T-Rex) — a complete pretraining stack for dexterous manipulation.

The Takeaway: See Actions, Then Do Actions

Internet vision taught robots to recognize the world. CAIP teaches them to read it for manipulation — attending to hands, contact regions, and motion-relevant objects rather than semantic categories alone.

For teams building on Sharpa Wave and the Dexmate / Isaac GR00T ecosystem, CAIP offers a concrete upgrade path: replace your VLA's frozen CLIP or DINO backbone with an encoder pretrained on 32,000 hours of human dexterity — and finetune on hundreds, not thousands, of robot demonstrations.

The eyes finally know what the hands are trying to do.