GALATEA takes its name from the sculpture brought to life in later tellings of the Pygmalion myth. The name captures the ambition of this work: turning motions imagined in generated videos into actions a robot can execute in the physical world.

A generated video may depict a hand lifting a jar or pushing a mug. Making that scene happen on a robot requires a controller that can coordinate the arm and fingers, establish contact, and move the object as intended.

Developed by researchers from UC Berkeley, Sharpa, and the University of Hong Kong, GALATEA connects generated manipulation videos to physical robot execution. It combines video generation, 3D hand–object reconstruction, and simulation training to build a unified dexterous controller, demonstrated on a Sharpa Wave hand.

Core Idea: Give Video Plans a Physical Execution Layer

GALATEA gives generated videos two roles: supplying motion references during training and proposing motion plans at deployment. A learned controller translates those references into robot actions.

This creates a useful division of responsibilities. The video describes the intended interaction; the controller must make it work through physical contact. Preserving both hand and object motion also retains information about how to approach and hold an object, alongside where to move it.

Technical Highlights: From Pixels to Contact

1. Generate an interaction from an image and an instruction. A real scene image and a language prompt condition the video model, making it possible to specify the object and intended behavior.
2. Recover motion in 3D. Stereo depth establishes metric scale. Hand and object tracking, followed by joint optimization, recover their trajectories and refine their relative placement.
3. Learn execution through simulation. Contact-preserving augmentation expands the references. Tracking rewards, contact and lift guidance, and the SAPG optimizer support reinforcement learning; domain randomization varies dynamics and observation noise.
4. Combine experts into one controller. Behavior cloning and DAgger distill category experts into a unified policy. During physical execution, visual object tracking supplies feedback for closed-loop control.

These components connect visual plans to an executable arm-and-hand policy. See the method in the paper.

Reconstruction Quality: A Blind Comparison

The researchers conducted a blind ranking study with four reviewers on 120 generated clips, producing 480 judgments per method. GALATEA ranked first in 417 judgments (87%). The mean ranks below summarize perceived reconstruction quality; lower is better. Paper, Section IV-A.

Experimental Results: Scale, Transfer, and Physical Execution

The project reports approximately 2,000 usable motion references from 2,500 generated clips, with more than 1,500 trajectories successfully grounded in simulation.

In the simulation evaluations:

• 77.4% mean success for GALATEA’s expert policies on the 10-object benchmark.
• 66.6% mean success for the distilled controller on 300 unseen trajectories.
• 54.2% mean success for the distilled controller on five novel objects, without retraining.

On the physical robot, the distilled controller completed 27 of 40 trials (67.5%) using unseen video plans. The four evaluated tasks were jar-neck pose adjustment, top-down jar grasping and moving, mug-rim pushing, and mug-handle pulling. These results demonstrate both grasping and manipulation without lifting the object. Real-world evaluation: Table III.

The Role of Sharpa Wave

1. A Human-Like Hand for Varied Interactions

Sharpa Wave has five fingers and 22 active degrees of freedom. This articulated structure offers researchers control over finger placement and hand shape when studying different ways to contact an object. For video-guided manipulation, that flexibility is especially relevant: the intended grasp is part of the task specification.

2. A Platform for Testing Learned Control in the Real World

GALATEA uses the Sharpa Wave hand mounted on a Franka Research 3 arm. Its physical trials put the full sequence to the test: approaching, making contact, and moving the object while receiving visual feedback. The jar demonstrations illustrate functional grasp choices, including holding the neck while leaving the opening accessible. See the setup and demonstrations.

Toward More Versatile Dexterous Robots

With demonstrations spanning grasping, pushing, pulling, and object-pose adjustment, GALATEA shows how generated video plans can become physical skills. Extending a shared controller to a wider range of interactions remains an important direction. The goal is to make the motions a video model imagines something a robot can reliably carry out.