As vision-language-action policies move from benchmarks to physical robots, evaluation is becoming a bottleneck of its own. A conventional test requires repeated hardware rollouts, manual scene resets, continuous monitoring, and binary success labels. The process is expensive to repeat, sensitive to small changes in object placement and contact, and often too coarse to explain why one policy behaves better than another.
Researchers from Sharpa and university collaborators introduced R2S-Eval, an evaluation pipeline that combines calibrated real-to-sim rollouts with vision-language model preference judgments. Instead of asking only whether a task eventually succeeds, R2S-Eval compares complete behaviors, including progress, smoothness, continuity, recovery, and visually observable control quality, then aggregates those pairwise judgments into statistically grounded policy rankings.
Real-to-sim calibration: the physical and simulated rollouts follow the same task definition, initialization range, observation and action interfaces, and success criterion. The objective is behaviorally useful correspondence, not a perfectly photorealistic digital twin.
Core Idea: Evaluate Behaviors, Not Just Outcomes
A binary score compresses an entire rollout into one bit. Two policies can both succeed even when one is smooth and direct while the other requires retries and unstable corrections. Two policies can also both fail even when one makes meaningful progress and the other barely moves. These differences matter when the goal is to understand capability rather than count terminal outcomes.
R2S-Eval reframes policy evaluation as preference estimation over rollout behaviors. Candidate policies generate video sets under a common task distribution. A VLM compares sampled pairs, and a Bradley–Terry model converts repeated preferences into policy-level scores and rankings. The resulting evaluator retains information that success rate discards while remaining compatible with conventional task metrics.
R2S-Eval in one view: candidate policies produce rollouts in a calibrated simulation, a VLM judges behavior pairs, pairwise preferences are aggregated into a ranking, and a validation protocol checks whether the conclusion is trustworthy.
Stage 1: Calibrate the Factors That Change Policy Behavior
R2S-Eval does not attempt to reproduce every visual detail of the laboratory. It calibrates the factors that determine policy execution, including robot geometry, kinematics, joint limits, control interfaces, task objects, camera viewpoints, initialization ranges, and task definitions. Candidate policies are then adapted with simulated data and deployed in closed loop in the calibrated environment.
This distinction is important. The pipeline does not assume that a simulation-side policy is identical to its real-world counterpart. It asks whether the comparative conclusion is preserved. In the seven tabletop tasks used in the study, six VLA policies were evaluated with 20 real-world trials and 200 simulation trials per policy–task setting. The real and simulated evaluations produced the same overall policy ranking, with a mean absolute success-rate difference of 2.13 percentage points across policy–task pairs and a maximum gap of 7.0 points.
Stage 2: Let a VLM Compare Complete Executions
For each rollout, the VLM first describes task progress, temporal continuity, motion smoothness, completion, and visible contact or effort control. It then compares two anonymized videos collected under the same task conditions and returns a binary preference. Randomized A/B ordering reduces position bias, balanced sampling prevents easy tasks from dominating, and repeated comparisons provide the evidence needed for ranking.
Behavior preference beyond success labels: both policies fail the same task, but one makes substantially more controlled progress. R2S-Eval is designed to preserve this execution-level difference instead of assigning both rollouts the same score.
The same principle applies at both ends of the outcome spectrum. Among two successful rollouts, a retry-heavy execution can be distinguished from a cleaner one; among two failures, partial progress and control quality still provide useful evidence.
Stage 3: Turn Preferences into a Validated Policy Ranking
Pairwise judgments are pooled into a preference matrix and fitted with a Bradley–Terry model. This produces a latent quality score for each policy and makes uncertainty measurable through bootstrap confidence intervals. More importantly, R2S-Eval validates the evaluator itself along five axes:
-
Reliability: Does the ranking agree with reference performance?
-
Stability: Do scores and ranks converge as more comparisons are collected?
-
Human agreement: Do VLM preferences align with how people compare the same videos?
-
Operator effort: Does the method reduce repeated hardware execution, reset, and monitoring?
-
Behavioral informativeness: Does it reveal progress, recovery, and control differences hidden by binary labels?
Experimental Results: Stable Rankings with Human Agreement
The evaluation covers 40 LIBERO tasks and seven calibrated real-world tasks, with six VLA policies and eight VLM judges. Three human annotators also evaluated 200 sampled video pairs in each setting, providing a majority-vote reference for preference agreement.
-
Hardware-consistent conclusions: In the calibrated real-to-sim setting, VLM rankings reached an average Spearman correlation of 0.957 with hardware success ordering, while preference scores reached a Pearson correlation of 0.978 with hardware success rates.
-
Strong human agreement: Recalculated from the eight per-judge entries in Table IV, pairwise VLM judgments averaged 90.9% agreement with human preferences in the calibrated real-to-sim setting.
-
Consistency across evaluators: The study tested eight VLMs across the Qwen3-VL, LLaVA-OV-1.5, and Gemma 3 families. The conclusion was not tied to a single judge architecture or model scale.
-
Information beyond success: Preference scores separate policies with similar completion rates when their execution quality differs, including retry behavior, partial progress, oscillation, and recovery.
-
Lower repeated hardware burden: Conventional operator time grows roughly linearly with tasks and trials. After calibration, R2S-Eval can expand routine evaluation without requiring a matching increase in physical rollouts.
Why SharpaNorth Matters to This Study
All real-world experiments were conducted on SharpaNorth, a dual-arm humanoid manipulation platform with two 7-DoF arms, two 22-DoF SharpaWave hands, an articulated neck and upper body, and a 63-DoF active configuration in the reported setup. Two head-mounted RGB cameras provide workspace views, while two wrist cameras capture close-range hand–object interaction.
The SharpaNorth evaluation platform combines head and wrist vision, dual dexterous hands, a shared tabletop workspace, and the everyday objects used across the seven calibrated tasks.
This platform makes the evaluation problem demanding in the right way. The policies must coordinate high-dimensional actions under visual uncertainty and contact variation, while the simulator must preserve the factors that change closed-loop behavior. The result is not a claim that simulation replaces hardware. Rather, hardware establishes and validates the target evaluation setting, while calibrated simulation carries the repeated rollout burden once policy conclusions have been shown to transfer.
Conclusion
R2S-Eval advances robot policy evaluation from repeated success counting toward automated, statistically stable, and quality-aware behavior assessment. Its contribution is not just a VLM judge or a calibrated simulator in isolation. The full pipeline connects efficient rollout generation, evidence-grounded pairwise comparison, statistical preference aggregation, and an explicit protocol for validating the conclusion.
Three key takeaways:
-
Robot evaluation should preserve behavior, not only outcomes. Complete rollouts reveal progress, recovery, smoothness, and control quality that binary success cannot capture.
-
Real-to-sim is useful when it preserves conclusions. The goal is not perfect visual replication, but a calibrated environment whose policy ranking remains consistent with hardware evidence.
-
Automated judges also need evaluation. Reliability, stability, human agreement, uncertainty, operator effort, and behavioral informativeness are all part of a credible evaluation pipeline.
Explore the R2S-Eval project page and read the preprint.
