What the model receives
Task instruction, tool descriptions, permitted object vocabulary, decision history and 512 × 512 RGB observations. Private object IDs, simulator state, teacher plans and answer keys are excluded.
06 / EVALUATION PROTOCOL
Same interface.
Observable consequences.
Models receive a goal and public interaction tools. They must decide what to inspect, what to do next and whether the task is actually complete.
How observations arrive
Navigation, scanning and search return eight uniformly sampled execution frames. State-changing actions return before-and-after frames. Direct execution retains up to 32 frames; the harness organizes publicly observed evidence in memory.
What counts as success
The agent must issue stop with all required goal predicates satisfied. Atomic, two-subtask and three-subtask episodes allow 24, 36 and 48 high-level calls, respectively, including QA answers.
What execution controls
Shared rule-based System 1 skills provide observation, A* navigation and state-changing manipulation. Controlled events can alter outcomes. This isolates high-level reasoning rather than evaluating low-level motor control.
From a template to an executable task
- SpecifyInstruction, roles, initial conditions, goal predicates, events and questions.
- Match & initializeFind a compatible layout; bind objects, place assets and set task-relevant states.
- ValidateCheck consistency, reachability and goal satisfaction through teacher execution with configured events enabled.
- Attach QAScreen questions against traces for valid activation and resolvable answers.
Eight ways to probe situated understanding
About 3,500 attached questions cover the execution cycle. Questions activate on configured actions, steps or milestones; answers are determined from the evaluated episode’s state, not copied from the teacher trace.
Target state
Is an object clean, open or activated?
Target relation
Is an object inside its container or on its intended surface?
Post-action outcome
Did the preceding action achieve its intended effect?
Active issue
Which issue or execution discrepancy remains unresolved?
Inserted obligation
What additional requirement was introduced?
Remaining requirement
Which task requirement is still unmet?
Mid-execution progress
Which goals hold at this execution point?
Composite progress
Which subtask is active or complete after a handoff?
QA uses dedicated answer-only turns with two to four choices and no additional physical action. Triggered but unanswered questions count as incorrect. Task metrics are weighted by task counts; QA metrics use their own question denominators.
10 / READING THE BENCHMARK
What the results do—and do not—show.
Does the policy see the teacher’s plan or simulator truth?
No. Privileged bindings and reference decompositions support construction and scoring. The evaluated policy receives instructions, permitted vocabulary, history and visual observations.
Why report both QA trigger rate and answer accuracy?
Different execution paths reach different question triggers. High accuracy on a small triggered subset cannot be interpreted as understanding the entire task. Both denominators matter.
Does the harness use a different model or extra training?
No. The comparisons use the same Qwen backbone with training-free memory, agenda management, verification and recovery. The harness makes additional inference calls, so gains should not be read as equal-compute comparisons.
What are the limits of the setup?
Rule-based manipulation abstracts away low-level physical control. Simulation results do not establish real-robot execution performance. The AgiBot protocol uses fixed reference histories and cannot assess consequences of the agent’s own actions.
How should the task counts and source versions be interpreted?
The reported bank has 450 ProcTHOR and 420 BEHAVIOR episodes. Each backend includes 50 two-subtask and 50 three-subtask composites. Tables retain their source grouping (routine/observation/failure/composite: 130/90/130/100 for ProcTHOR and 100/90/130/100 for BEHAVIOR). The appendix uses a different semantic grouping; scores here are not regrouped. The newer container-inspection description should not be read as a newly measured score for the legacy native-QA template.