Skip to content
Artwork for Embodied AI 101
Embodied AI 101 · Yesterday · 12 min

HexaShell: Can Debugging Code Generalize RGB Robot Manipulation?

Can a robot improve at manipulation by debugging code instead of updating neural-network weights? This episode explains how HexaShell turns robot experience into reusable tools, and what its RoboDojo score actually establishes. We cover three questions: how calibrated RGB observations become metric positions; why the reported 39.7% success rate uses category-balanced scoring; and how holding out layouts differs from holding out tasks. The experiments use a simulated dual-arm ARX X5 with head and wrist cameras, camera calibration, and proprioception. Along the way, we connect the design to Code as Policies and URAI, examine an inspectable geometric helper, and separate tool development from execution with frozen tools. The central limitation is a mismatch in task-level learning conditions: HexaShell develops tools on tasks in a category that RoboDojo's published training split reserves for transfer. For researchers building language-driven manipulation systems, the episode offers a concrete way to think about code as learned policy state. If software is part of what a robot learns, where should the training boundary be drawn — and what would convincing generalization look like?

0:00-12:06

transcript

No transcript — this publisher did not publish one.

show notes

Can a robot improve at manipulation by debugging code instead of updating neural-network weights? This episode explains how HexaShell turns robot experience into reusable tools, and what its RoboDojo score actually establishes. We cover three questions: how calibrated RGB observations become metric positions; why the reported 39.7% success rate uses category-balanced scoring; and how holding out layouts differs from holding out tasks. The experiments use a simulated dual-arm ARX X5 with head and wrist cameras, camera calibration, and proprioception. Along the way, we connect the design to Code as Policies and URAI, examine an inspectable geometric helper, and separate tool development from execution with frozen tools. The central limitation is a mismatch in task-level learning conditions: HexaShell develops tools on tasks in a category that RoboDojo's published training split reserves for transfer. For researchers building language-driven manipulation systems, the episode offers a concrete way to think about code as learned policy state. If software is part of what a robot learns, where should the training boundary be drawn — and what would convincing generalization look like?