Propose explanations
A vision-language model uses the demonstrations and scene states to propose Python programs specifying object roles, goals, ordered subgoals, and constraints.
Rational Inverse ReasoningFew-Shot Imitation by Inferring
Programmatic Intent through Planning
Supplementary material
A few demonstrations show what happened.
RIR infers why it happened, then plans what to do in a new scene.
Abstract
Humans learn a manipulation task from one or two demonstrations, and perform it in a new room, with new objects, under new constraints. This requires identifying which object roles, subgoal orderings, and constraints should persist when the scene changes, as opposed to cloning object motions. We present Rational Inverse Reasoning (RIR), which infers this task structure as an explanation program: a compact, executable description of intent that maps an object-centric scene to a task-and-motion planning (TAMP) specification of goals, subgoals and constraints. This allows the robot to generate new actions for new objects and layouts, while preserving the demonstrator’s intent.
A vision-language model proposes candidate programs, a planner-derived score evaluates whether each candidate makes the demonstrated behavior rational, relative to alternative strategies. These scores guide iterative program refinement. We evaluate RIR on 30 simulated, and four real-world tabletop tasks (using a Franka FR3), and 35 two-dimensional reasoning tasks. In all cases, RIR recovers transferable task structure from only 1–3 demonstrations. Generalizing to new layouts and object sets, it outperforms VLM, planning and few-shot learning baselines that lack explicit rationality and planning-grounded inference.
Method
Infer the intent. Re-plan the motion.
An explanation program captures what should stay the same when the objects, layout, and geometry change.
Reason with a planner
Compare demonstrated and alternative strategies. Planner-derived scores and diagnostics guide revisions to a finite pool of candidate explanations.
Generalize through planning
Run the selected program on a new scene, re-ground its goals and constraints, and plan a new sequence of actions that preserves the inferred intent.
Why did the demonstrator choose this behavior?
Several programs can fit the same demonstration. RIR asks whether the observed strategy is a sensible choice under each explanation, whether its motion is feasible and efficient, and whether its constraints explain the behavior’s specificity.
Evaluation
Learning from 1–3 demonstrations
From continuous robot manipulation to geometric and algorithmic reasoning.
30simulated tabletop tasks
4real-world task families
352D reasoning tasks
Simulated manipulation
Same intent, different scene
Thirty tabletop tasks in IsaacSim span more than 300 objects and 90 held-out scenes. Tasks require object selection, ordered subgoals, and geometric constraints, with changes in object identity, count, layout, and clutter.
RIR reports 1.4× comprehension and 1.3× overall success relative to its nearest baseline, Reasoning without Rationality (Rw/oR), across the one- and three-shot settings.
Rw/oR uses the same VLM and proposal budget, with self-refinement in place of planner-derived rational feedback.
Simulated task distribution
Explore the task environments
Thirty manipulation tasks, with changes in objects, layouts, and clutter. Select a task to explore three training environments and five examples from its evaluation distribution.
Fly-arounds show the initial environments. They do not depict task execution.
Real-world manipulation
From teleoperation to new tabletop scenes
A Franka FR3 learns long-horizon tasks from teleoperated demonstrations. A real-to-sim pipeline reconstructs object-centric scenes so the inferred programs can transfer to new objects, layouts, and distractors.
75%
Intent comprehension
With one or three demonstrations, measured as program equivalence.
42% → 57%
Trajectory success
One to three demonstrations on held-out scenes; three-shot Rw/oR achieves 10%.
13 / 20
Physical task completion
A separate execution case study on a learned explanation program.
2D reasoning
Recover spatial and algorithmic structure
The 35-task corpus includes 25 spatial tasks and 10 additional tasks involving filtering, counting, and conditionals. Even simple demonstrations can admit several explanations, making both proposing and ranking the intended program important.
| Task group | Rw/oR | RIR |
|---|---|---|
| Spatial | 22.4% | 58.9% |
| Spatial + algorithmic | 14.5% | 24.0% |
How to interpret the results
Three metrics answer different questions. Comprehension measures whether the program recovers the intended task; trajectory success checks planned behavior in a held-out scene; execution success measures task completion during a physical or simulated rollout.
Current limitations. RIR uses an approximate, planner-dependent score and searches a finite pool of VLM-generated programs. Perception and planning can fail, and real-robot plans currently execute open-loop. In the physical case study, missed grasps caused four of seven failures.
See Sections IV–V and the appendices of the paper for the full protocol and limitations. Numerical summaries on this page follow the supplied manuscript.
BibTeX
Citation for the anonymous manuscript.
@misc{anonymous2026rir,
title = {Rational Inverse Reasoning: Few-Shot Imitation
by Inferring Programmatic Intent through Planning},
author = {Anonymous Authors},
year = {2026},
note = {Anonymous manuscript}
}