OmniLinkExplore OmniSim ↗OmniLink in OmniSim · Development benchmark · September 24, 2026
OmniLink completed all 33 tasks in OmniSim in this run, at 45% lower model cost per successful task than the tested plain model loop. Explore every configuration, the limits of the comparison, and the evidence behind it.
01 / Completion and cost
One run per task and configuration. These are controlled integrations built for the benchmark, rather than complete hosted products or every possible configuration of the frameworks.
USD · all attempts included · bars start at zero
All configurations use the same model and caching policy. Shared warm-up cost is reported separately.
| Configuration | Completed | Failed | Unwanted motion | Model calls | Model USD | USD / success |
|---|---|---|---|---|---|---|
| OmniLink | 33/33 | 0 | 0 | 40 | $0.498418 | $0.01510 |
| Plain model loop | 32/33 | 1 | 0 | 71 | $0.879256 | $0.02748 |
| LangGraph | 33/33 | 0 | 0 | 72 | $0.884684 | $0.02681 |
| Lobster SDK | 33/33 | 0 | 0 | 72 | $0.901300 | $0.02731 |
| LangGraph + OmniLink command handling | 33/33 | 0 | 0 | 41 | $0.492716 | $0.01493 |
| Lobster SDK + OmniLink command handling | 33/33 | 0 | 0 | 40 | $0.488451 | $0.01480 |
No infrastructure errors. Total model cost: $4.150898, including $0.006072 for two shared cache warm-ups. All 338 paid requests have recorded usage. Model costs use provider list rates; subscriptions, hosting and local compute are excluded.
Cost ratio = OmniLink model cost per success ÷ comparator model cost per success; below 1 favors OmniLink. Intervals use 2,000 bootstrap draws over 19 task families, retaining robot variants together. They describe this authored suite and do not establish a market-wide ranking.
| OmniLink versus | Completion difference | Cost ratio | 95% cost interval |
|---|---|---|---|
| Plain model loop | +3.03 pp | 0.550 | 0.314–0.832 |
| LangGraph | +0.00 pp | 0.563 | 0.321–0.875 |
| Lobster SDK | +0.00 pp | 0.553 | 0.314–0.859 |
| LangGraph + OmniLink command handling | +0.00 pp | 1.012 | 0.943–1.106 |
| Lobster SDK + OmniLink command handling | +0.00 pp | 1.020 | 0.983–1.078 |
Allocating one-sixth of warm-up to each configuration changes OmniLink to $0.01513 per success and the plain loop to $0.02751. Exact values for every configuration are in results.json.
02 / Method and scope
The 33 development tasks cover motion, pose reporting, follow-up instructions, cancellation, ambiguous and quoted commands, conditions, and injected failures. Robots: Husky, TurtleBot3 Burger and OmniArm 6. Arm grasping uses a simulator-assisted attachment.
Every configuration receives the same instructions, tools, measured state, history, controllers and safeguards. Each episode starts in a fresh simulator session. The shared limits are six planning rounds and 24 actions per operator turn.
Completion is based on measured positions, ordered waypoints, joint angles, object handling and required response status. A model saying “done” does not establish success. Motion is sampled at 30 ms wall intervals and tool boundaries, with explicit tolerances.
Cost per success includes model usage from failed tasks. All configurations use five-minute caching with common prefixes warmed before scoring. This shared-workspace cache setup does not measure isolated production cache behavior.
Model: claude-opus-5-5, direct Anthropic API, low effort, standard tier, 2,048 maximum output tokens including thinking. No paid-request retries. Rates per million tokens: ordinary input $4, output $20, five-minute cache writes $5 and cache reads $0.20. Usage is derived from returned counters, not reconciled to an invoice.
Machine: Windows 11; AMD64 Family 25 Model 80, 16 logical processors; NVIDIA RTX 3060 Laptop GPU. Physics used Newton 1.5.0 / MuJoCo 3.11.0 on CPU; GPU acceleration was not used for benchmark physics. Machine fingerprint: 9722d23d12a3. Python 3.12.14; Node 22.19.0; LangGraph 1.2.12; pinned Lobster SDK commit 71bd8145055498116de35623dbea0d7dda9b4cd5. Complete hashes are in the archived lock.
Order was shuffled with fixed seed 240926. The suite and configuration were frozen before this campaign. The tasks were previously seen during development; they are not an independent holdout. OmniLink names the tested agent-control integration; OmniSim is the simulator used for every configuration. This does not measure every feature of either product.
LangGraph uses its actual graph runtime; Lobster uses its actual SDK pipeline. Their planning/execution callbacks share benchmark logic. The two “+ OmniLink command handling” configurations are component-sharing controls. Exact mappings to the original record identifiers are in results.json; evidence and source are preserved without relabeling.
The plain model loop did not finish a TurtleBot3 conversation task. It requested clarification instead of executing the final instructed motion after detecting a small earlier position discrepancy. The frozen completion criteria marked it failed. No unwanted motion was observed. The episode remains in every denominator and in the raw evidence.
This developer-authored suite contains one repeat per task. It excludes vision-based navigation, fleet coordination, autonomous exploration and physical hardware. Sampled “no unwanted motion” observations are not continuous safety monitoring. The results establish neither superiority over all competitors nor a general capability lead.
03 / Evidence and reproduction
The evidence archive preserves raw responses, provider usage, simulator observations, physics logs, task definitions, frozen source and the verification program. All 198 grades and 1,143 file hashes passed offline verification.
Extract evidence.zip into a new folder, open a terminal there, and run:
python -I -S verify_archive.pyNo API key, network, simulator or paid call is needed. The verifier recomputes grades and costs from saved observations and checks archive hashes. Hashes establish consistency; this is not independent validation of the original run.
Offline verification is available now. Running the robots and model again requires the matching OmniSim 9.0.0 Windows runtime, source, assets and pinned dependencies, plus your own OmniKey and Anthropic API key. The archive omits the engine binaries; public availability of the exact matching release has not been verified. Do not substitute an older public build or assume a version number alone guarantees a match. Check the archived hashes. Live runs incur provider charges and may differ as the hosted model changes.
The earlier Gemini campaign completed 198 episodes with 32/33 completions and one unwanted-motion episode for every configuration. It used a different model transport and prompt wrapper, so it is historical evidence rather than a controlled model-only comparison.
An uncached Opus run was stopped at the owner's request to prepare this cached comparison: 101 completed episodes, $3.397460 in known request costs, plus one unresolved in-flight reservation of up to $0.085648. It is incomplete and is not pooled with this run. The cached run started from a new frozen campaign and completed all planned episodes. Earlier pilots and failures are retained in the history archive; any privacy redactions are listed with original and exported hashes.
The campaign protocol's “prepared, not started” wording records its pre-run state. It is retained unchanged; completed results and timestamps are in the final evidence. Historical report statements about no deployment describe the measurement/export step, not the publication status of this website.
Next research milestone · planned
Our next step is a harder, independently reviewed evaluation of recovery from disturbances and partial execution, using unfamiliar tasks and strong complete-system baselines. That work is planned; no capability result is claimed here.
Explore OmniSim ↗