Bridging Task Planning and Execution for Language-Conditioned Bimanual Manipulation

TL;DR — A zero-shot VLM framework for bimanual manipulation in clutter that reasons not only about what objects are, but about how they can physically be reached. Each execution stage is validated against explicit pre- and post-conditions, so that failures are recovered locally whenever possible and the plan is revised only when needed.

Abstract

Robots are increasingly expected to execute natural-language manipulation tasks in real-world environments. This is particularly relevant in household assistance and logistics, where clutter requires reasoning about both task objectives and execution feasibility. This challenge becomes even more pronounced for bimanual humanoid platforms, where coordinating two manipulators introduces additional interaction and kinematic constraints.

We present a zero-shot vision-language framework for execution-aware bimanual manipulation that jointly reasons about task objectives and physical feasibility. Planning is grounded in a structured scene representation that combines semantic object relations with execution constraints, enabling the selection of obstacle relocations and execution choices while preserving task semantics. Compatible actions are organised into parallel bimanual stages with explicit pre- and post-conditions. A hierarchical validator evaluates these conditions under perceptual uncertainty and uses stage-level execution history to support local recovery or targeted replanning.

Experiments on the CENTAURO hybrid wheeled-legged humanoid robot across nine language-conditioned manipulation scenarios show higher task success than an execution-agnostic baseline. Dedicated failure scenarios further show that structured validation feedback supports effective recovery from execution failures. These results demonstrate the benefits of combining execution reasoning directly into task planning.

Why Execution Awareness?

Boxes Aside, instruction “Grasp the white box”. Left: execution-aware planner. Right: planner using only symbolic object relations.

Symbolic relations such as right of describe where objects are, but not whether the robot can actually reach them. When a box stands right next to the target, a planner that only sees these relations does not recognise it as an obstruction and attempts the grasp directly.

Our scene representation annotates every side of every object with its accessibility (is there enough free space for the end-effector?) and reachability (is there a kinematically feasible approach?). The planner can then detect the blocking object and relocate it first.

Boxes Aside success rate: 0% → 85%

Method

Overview of the pipeline: scene perception, manipulation planning, scheduling into parallel stages, and validation.

Pipeline overview. From an RGB observation and object poses and dimensions, the framework constructs a structured scene representation, generates a sequential manipulation plan, and groups compatible actions into parallel execution stages. The validator checks stage-level pre- and post-conditions and maintains a history of execution attempts to guide validation and recovery.

1Structured scene representation

A VLM identifies objects, attributes (e.g., fragility), pairwise relations, and the state of the end-effectors. Object poses ground the graph geometrically: each side of every object is labelled as accessible or blocked (and by what), and as reachable or not by each arm.

2Execution-aware manipulation planning

A VLM turns the instruction into a sequence of actions specifying the primitive, the target, the arm(s), whether the claw is engaged, and the side from which to interact. When the target is not directly graspable, the blocking objects are relocated first.

3Parallel bimanual scheduling

Compatible actions are grouped into stages executed concurrently by the two arms. Each stage carries a local goal and explicit pre- and post-conditions, expressed as atomic, independently verifiable predicates.

4Hierarchical validation & recovery

Each condition is classified as satisfied, uncertain, or violated. Uncertain outcomes trigger additional perception; confirmed failures are interpreted using the scene transition and the attempt history, and resolved from local recovery to global replanning.

Recovery in Action

Failures are deliberately injected during execution. The three strategies re-invoke progressively larger portions of the pipeline.

Repeat

Bottle Failure

Execution made progress but did not complete: the scene is perceived again and the stage is re-executed unchanged.

Modify

Push Failure

No progress, but the failure is attributable to an action attribute: an Action Revision VLM revises the arm, contact, or side, and the stage is re-executed.

Replan

Stacked Boxes w/ Failure

The stage is no longer applicable: planning and scheduling are re-invoked from the current scene, taking the reported failure into account.

Example Execution

Execution timeline of the Fragile Box scenario: push of the blue box, placement of the fragile green box on the table, and final grasp of the white box.

Fragile Box. The task is to grasp the white box. The robot identifies the blue and green boxes as obstacles placed on top of the target. Since they are laterally too close to be grasped directly, the robot first pushes the non-fragile blue box from its accessible front side, the most suitable direction for clearing the target. It then grasps and places the fragile green box on the table. Once both obstacles have been removed, the robot grasps the white box.

Experimental Scenarios

Nine tabletop scenarios on the CENTAURO platform, each evaluated over 20 executions. The robot receives only a short instruction, without information about the obstacles or the required manipulation strategy.

Execution-aware planning

Stacked Boxes: A box rests on top of the target.
Boxes Aside: Objects beside the target block its lateral sides.
Bottles on Box: Two bottles rest on top of the target.
Fragile Box: Obstacles on the target include a fragile box.
Long Horizon: Several stacked and adjacent obstacles must be cleared.

Dynamic scene adaptation

Scene Update: Place the ball on the blue box while keeping the top of the white box clear; a bottle placed on the white box during execution triggers replanning.

Execution recovery

Bottle Failure: Pick up the bottle under an injected execution failure.
Push Failure: Move the red box away from the white box under an injected push failure.
Stacked Boxes w/ Failure: The grasp of the obstacle is systematically prevented.

Results

We compare three configurations sharing the same VLM backbone, instructions, primitives, and low-level execution: a planner that sees only symbolic object relations with a binary success/failure validator, the execution-aware planner with a binary validator, and the full framework with hierarchical validation. SR is the percentage of the 20 executions completed successfully; Queries is the average number of VLM queries per successful execution.

Scenario	Non-execution-aware & binary validation		Execution-aware & binary validation		Execution-aware & hierarchical validation (ours)
Scenario	SR	Queries	SR	Queries	SR	Queries
Execution-aware planning
Stacked Boxes	100%	8.2	100%	8.2	100%	8.2
Boxes Aside	0%	–	75%	12.3	85%	11.8
Bottles on Box	70%	9.6	90%	9.4	95%	9.4
Fragile Box	0%	–	70%	11.7	70%	11.5
Long Horizon	0%	–	35%	24.3	50%	21.1
Dynamic scene adaptation
Scene Update	85%	14.6	85%	14.6	85%	14.4
Execution recovery
Bottle Failure	90%	10	90%	10	90%	8.1
Push Failure	15%	10	85%	10	95%	9.1
Stacked Boxes w/ Failure	90%	12	90%	12	90%	12

Execution awareness matters when obstruction is lateral. With side-by-side objects (Boxes Aside, Fragile Box, Long Horizon), symbolic relations cannot tell whether the free space suffices for the end-effector, and the non-execution-aware planner fails in all executions. Encoding side-level accessibility and reachability raises success to 85%, 70%, and 50%.
Hierarchical validation makes recovery cheaper. A binary validator must always replan after a failure (3 queries), whereas repeat and modify require 1 and 2 queries. This reduces the average queries from 10 to 8.1 in Bottle Failure and from 10 to 9.1 in Push Failure.
Parallel scheduling shortens execution. Grouping compatible actions into bimanual stages reduces robot execution time by 40% in Bottles on Box and by 11% in Long Horizon, while preserving task completion.

BibTeX

@inproceedings{cuzzocrea2026bridging,
  title     = {Bridging Task Planning and Execution for Language-Conditioned Bimanual Manipulation},
  author    = {Cuzzocrea, Emanuele and Iannotti, Giuseppina and Rossini, Luca and Laurenzi, Arturo and Tsagarakis, Nikolaos},
  booktitle = {IEEE-RAS International Conference on Humanoid Robots (Humanoids)},
  year      = {2026}
}