When a robot misses the stack, which model needs fixing?

Suppose a robot reaches for the right block, turns it the wrong way, and misses the target stack. The episode gets a failure mark.

That mark doesn't tell you whether the robot chose a bad instruction or failed to execute a useful one. A hierarchical robot has both a high-level planner, which chooses the next skill and target, and a low-level controller, which moves the arm.

Ai2's new ToyBlocks release makes this split testable: either policy level can be replaced by a reference component, called an oracle, while evaluating tasks in the simulated world. [1][2] The useful idea is the controlled replacement, not just another overall success percentage.

Computed independent-stage illustration. These are not robot trials, ToyBlocks measurements or named-model scores. Image by the author.

A tie can come from different pipelines

Here's a deliberately simple model. A task succeeds only when the planner produces a valid plan and the controller executes that plan successfully.

Write the success probability as:

P(success) = P(valid plan) × P(success | valid plan)

In this construction, an invalid plan always fails. System A has a valid-plan probability of 0.9 and an execution probability of 0.6 given a valid plan. System B reverses those numbers.

Both finish at 0.54. The final score alone cannot identify which stage is weaker.

Now imagine a perfect replacement for one stage, holding the other stage's probability fixed. Replacing A's planner gives 0.6; replacing its controller gives 0.9. For B the effects reverse.

Those are assumptions about an invented pipeline. In a real system, changing the planner can also change which instructions or states the controller encounters. The remaining stage's behavior may change too.

Enumerate the little model

I used a Cartesian product with ten equally likely outcomes for each stage. That gives 100 equally weighted combinations, not 100 sampled robot episodes.

This standard-library Python program checks every combination and independently verifies each result with exact fractions. Save it as stages.py and run python3 stages.py:

from itertools import product from fractions import Fraction # An invented independent-stage model with 100 equally weighted outcomes. outcomes = list(product(range(10), repeat=2)) def success(planner_slots, controller_slots): count = sum(p < planner_slots and c < controller_slots for p, c in outcomes) rate = Fraction(count, len(outcomes)) assert rate == Fraction(planner_slots, 10) * Fraction(controller_slots, 10) return count for name, planner, controller in [('A', 9, 6), ('B', 6, 9)]: print(f'{name}: original={success(planner, controller)}/100 ' f'perfect-planner={success(10, controller)}/100 ' f'perfect-controller={success(planner, 10)}/100')

Output:

A: original=54/100 perfect-planner=60/100 perfect-controller=90/100 B: original=54/100 perfect-planner=90/100 perfect-controller=60/100

I also checked the product calculations separately. The code demonstrates why a tied pipeline score is ambiguous. It does not evaluate a robot, run a simulator or establish the effect of an oracle in ToyBlocks.

Turn the replacement into an actual test

ToyBlocks supplies more of the machinery needed for that next step. Its evaluation can start from a recorded episode's scene and task definition, including the robot pose, block poses, geometry and goal. It supports learned high-level commands with a learned low-level policy, a symbolic high-level planner, or reference low-level skills used in data generation. [2]

An oracle here is a concrete reference implementation. It should not be assumed physically infallible just because the toy arithmetic used a perfect stage.

The task's success signal also matters. The documentation describes checking the logical goal, settled goal blocks and an open gripper, with partial progress derived from goal predicates. [2] A training reward and a completed task are different measurements.

For an oracle swap, I would keep the episode definitions and success rule fixed, compare the complete system with each replacement, and inspect the failed episodes as well as the aggregate. A better result after a replacement is evidence about that tested setup; it doesn't automatically assign every failure to one component.

Preserve the conditions around the score

The evaluation documentation explicitly aims for statistical rather than bit-for-bit reproducibility. Shared physics, GPU solving and scheduling can change individual trajectories even when episode inputs and seeds are fixed. [2]

That makes a saved initial scene, scoring configuration and per-episode record more useful than a screenshot of one successful stack.

Scchy's MicroDuck training write-up emphasizes inspecting actual skill rollouts. [3] ToyBlocks adds a complementary question for a hierarchy: what changes when one part is replaced while the task definition stays put?

I read the released framework and evaluation design; I haven't run its GPU simulation or reproduced its model results. The worked example is a way to understand the diagnosis before interpreting a robotics score.