Robot Demo Guide

1. Overview

The robot demo places a robot on a 20×15 grid (300 cells) with obstacles, collectible items (gems, stars, coins, crystals), fog of war, and a battery that drains as the robot moves. You control the robot by typing natural language in the OmniLink web UI — the AI translates your requests into structured tool calls that execute on the simulation.

Architecture

  OmniLink Web UI              OmniLink Platform
  (browser / phone)    HTTPS    (AI engines with native tools)
        |            <------>          |
        |              /api/chat        |  Engine sends tools via
        |                               |  provider's native API:
        |                               |   - OpenAI: tools[].function
        |  HTTP (toolCallbackUrl)       |   - Anthropic: tools[].input_schema
        v                               |   - Gemini: functionDeclarations
  ToolRunner                            |
  (run_tools_demo.py)    HTTP     <-----+
  - Tool HTTP server   <------>
  - Profile setup        POST /tool
  - Activity log
        |
        |  HTTP (localhost:5050)
        v
  robot_sim.py
  (Pygame + HTTP server)
  - 20x15 grid world
  - Actions, state, rendering
  - Battery, score, fog of war

Three Components

2. Quick Start

Prerequisites

Terminal 1 — Start the Simulation

python -m omnilink.examples.robot_demo.robot_sim

This opens a Pygame window showing the 20×15 grid with the robot, obstacles, items, and fog of war. An HTTP server starts on port 5050.

Terminal 2 — Start the ToolRunner

OMNI_KEY="olink_..." python -m omnilink.examples.robot_demo.run_tools_demo

On Windows (PowerShell):

$env:OMNI_KEY = "olink_..."
python -m omnilink.examples.robot_demo.run_tools_demo

The ToolRunner connects to the simulation, registers the agent profile with 16 tool definitions, starts a tool HTTP server on a dynamic port, and waits for requests.

Open the Web UI

Open https://www.omnilink-agents.com/build in your browser. Select the robot-demo agent and start chatting. Try messages like:

• move forward / turn left / scan the area
• where are you? / what's nearby? / check your status
• go to position 10,5 / pick up the item
• explore the map and collect all items
• find a path to 15,12

3. Architecture — Native Structured Tool Calling

Every interaction uses the same flow. The AI calls tools using the provider’s native structured tool calling API — not text-based Tool: tags. The frontend dispatches tool calls via HTTP without any regex parsing.

The Full Pipeline

  1.  User types "move forward" in the web UI
  2.  UI sends to /api/chat with systemInstructionRequest
      (includes all 16 tool definitions with JSON Schema parameters)
  3.  Engine converts tool definitions to the provider's native format:
        - G2 (OpenAI/GPT):  requestPayload.tools[].function
        - G3 (Grok/xAI):    requestPayload.tools[].function  (OpenAI-compatible)
        - G4 (Claude):       requestPayload.tools[].input_schema
        - G1 (Gemini):       config.tools[].functionDeclarations
  4.  Provider responds with a structured tool call (not text):
        - OpenAI/Grok: message.tool_calls[{id, function.name, function.arguments}]
        - Claude:      content[{type: "tool_use", id, name, input}]
        - Gemini:      parts[{functionCall: {name, args}}]
  5.  Engine normalizes to universal ToolCall format:
        {id: string, name: string, arguments: Record}
  6.  API response includes: {text: "...", toolCalls: [{id, name, arguments}]}
  7.  Frontend receives structured toolCalls — no regex parsing needed
  8.  Frontend POSTs to toolCallbackUrl: {"tool": "move_forward"}
  9.  ToolRunner's HTTP server dispatches to the tool's execute() method
  10. Tool calls robot_sim via HTTP, returns result JSON
  11. Frontend sends tool result back to AI for follow-up response
  12. AI responds with summary or next tool call
  13. Loop continues until no more tool calls (up to maxToolRounds)

How Tool Definitions Flow

  Python (DefaultTool.to_dict())
    {"name": "go_to", "description": "...",
     "parameters": {"type": "object", "properties": {...}}}
         |
         v  Stored in agent profile on OmniLink platform
         |
  chat.ts (resolveSystemInstruction)
    Merges profile settings with frontend request
    Passes availableToolDetails to engine handler
         |
         v
  Engine handler (e.g. g3-engine.ts)
    extractToolDefinitions() -> ToolDefinition[]
    Converts to provider-native format
         |
         v  Sent in API request to provider
         |
  Provider responds with structured tool call
         |
         v
  Engine normalizes to universal ToolCall {id, name, arguments}
         |
         v  Returned in API response as toolCalls[]
         |
  Frontend (handleToolTagsFromResponseInternal)
    Prefers structured toolCalls, falls back to text parsing
    POSTs {tool: name, ...args} to toolCallbackUrl
         |
         v
  ToolRunner HTTP server (/tool)
    Dispatches to DefaultTool.execute(**kwargs)
    Returns JSON result

Text-based fallback: If the provider does not return structured tool calls (e.g. older models), the frontend automatically falls back to regex-parsing Tool: name({"arg": "val"}) from the response text. This requires no configuration.

4. The 16 Tools

All 16 tools are DefaultTool subclasses defined in robot_tools.py. Each has a name, description, parameters_schema (JSON Schema), and an execute(**kwargs) method.

Movement Tools

Action Tools

Query Tools

5. Creating Your Own Tools

Every tool is a subclass of DefaultTool from omnilink.default_tools. You define four things: the tool name, a description for the AI, a parameters_schema (JSON Schema), and an execute(**kwargs) method that does the work.

The DefaultTool Pattern

from omnilink.default_tools import DefaultTool

class MyTool(DefaultTool):
    name = "my_tool"
    description = "Does something useful."
    parameters_schema = {
        "type": "object",
        "properties": {
            "target": {"type": "string", "description": "The target."},
        },
        "required": ["target"],
    }

    def execute(self, **kwargs):
        target = kwargs["target"]
        # ... do work ...
        return {"status": "done", "target": target}

For tools with no arguments, use an empty schema:

parameters_schema = {"type": "object", "properties": {}}

Important: Every tool must have a parameters_schema. OpenAI-compatible APIs require the parameters field to recognize function definitions. Omitting it will cause provider-level errors.

Full Example: GoToTool

from omnilink.default_tools import DefaultTool
from .robot_api import get_state, send_action

class GoToTool(DefaultTool):
    name = "go_to"
    description = (
        "Navigate the robot to a specific grid position using A* pathfinding. "
        "Moves the robot along the full path in one call. "
        "Args: x (required), y (required)."
    )
    parameters_schema = {
        "type": "object",
        "properties": {
            "x": {"type": "integer", "description": "Target x coordinate."},
            "y": {"type": "integer", "description": "Target y coordinate."},
        },
        "required": ["x", "y"],
    }

    def execute(self, **kwargs):
        x = kwargs.get("x")
        y = kwargs.get("y")
        if x is None or y is None:
            return {"error": "Missing required arguments: x and y"}
        msg = send_action("go_to", x=int(x), y=int(y))
        state = get_state()
        r = state.get("robot", {})
        return {
            "status": "navigating",
            "target": {"x": int(x), "y": int(y)},
            "position": {"x": r.get("x"), "y": r.get("y")},
            "battery": state.get("battery"),
            "message": msg,
        }

Registering Tools in a ToolRunner

Create a ToolRunner subclass and assign your tools to the default_tools class attribute:

from omnilink.tool_runner import ToolRunner
from .robot_tools import ALL_ROBOT_TOOLS

class RobotToolsRunner(ToolRunner):
    agent_name = "robot-demo"
    display_name = "Robot Demo"
    engine = "g3-engine"
    tool_name = "control_robot"
    tool_description = "Control the robot using the available tools."
    commands = "stop_game, pause_game, resume_game"

    default_tools = ALL_ROBOT_TOOLS

    def get_profile_settings(self):
        settings = super().get_profile_settings()
        settings["mainTask"] = (
            "You control a mobile robot on a grid world. "
            "Use the available tools to move the robot, collect items, "
            "and explore the map."
        )
        settings["maxToolRounds"] = 5
        return settings

    def get_state(self):
        return {}

    def execute_action(self, state):
        pass

    def state_summary(self, state):
        return "idle"

    def is_game_over(self, state):
        return True

The ToolRunner automatically converts each DefaultTool into an availableToolDetails entry (via DefaultTool.to_dict()), registers the agent profile, starts the tool HTTP server, and handles incoming tool calls by dispatching to the correct tool’s execute() method.

6. Multi-Engine Support

All four OmniLink AI engines support native structured tool calling. Each engine converts tool definitions to the provider’s native format and normalizes the response to a universal ToolCall interface:

interface ToolCall {
  id: string;                         // Provider-assigned or generated UUID
  name: string;
  arguments: Record<string, unknown>;
}

Engine-to-Provider Mapping

Single-Turn Test Results (20 prompts per engine)

Key findings:

• All engines correctly select tools from the 16 available options.
• Parameter extraction (coordinates, speed values) is accurate across all engines.
• G2’s 2 “partial” results were multi-step prompts where only the first tool was issued (expected in single-turn without tool-result loop).
• G3 and G4 support parallel tool calls (multiple tools per response).

7. Performance Benchmarks

Benchmarked using the full UI pipeline (/api/chat) with the ToolRunner HTTP server executing tools on the Pygame simulation. Grid: 20×15 (300 cells), 54 obstacles, 10 items, up to 50 tool rounds per task.

go_to Optimization: Before vs. After

The go_to tool was upgraded from single-tile movement to full A* pathfinding in one call. This eliminated the need for dozens of AI round-trips to traverse the map:

Root cause: Each go_to call required a full AI round-trip (~5–10s). Moving one tile at a time meant 20+ round-trips for cross-map navigation. With full-path go_to, the robot traverses the entire A* path in a single call.

Long-Horizon Exploration (v2, G3 Engine)

v2 Aggregate

Key Observations

1. Full-path go_to is transformative. Reducing navigation from many single-tile calls to one full-path call cut tool calls by 17x and duration by 10x.
2. Long-horizon planning works. The AI sustains coherent multi-round strategies, adapting to obstacles and collecting items opportunistically.
3. Smart early stopping. When a goal is already met, the AI stops instead of wasting tool calls.
4. Parallel execution. The AI calls 5–7 tools per round, gathering information in bulk before acting.
5. No tool hallucination. Across 264 total tool calls (v1 + v2), the AI never called a non-existent tool or passed malformed arguments.
6. Battery is the real constraint. With full-path navigation, the robot covers much more ground but drains battery faster.

8. Known Issues & Fixes

Issues discovered and resolved during the native tool calling migration. Documented here for future reference.

Frontend not sending tool definitions

Symptom: AI responds “my navigation systems are offline” instead of calling tools.
Cause: The frontend’s getAvailableToolDetails() and formatAvailableTools() were hardcoded stubs returning empty arrays. The /api/chat endpoint received no tool definitions, so the AI had no tools to call.
Fix: Load availableToolDetails and availableTools from the agent profile settings and return them from these methods.

Frontend overwriting profile tool definitions

Symptom: Tools work once after starting the ToolRunner, then disappear on page reload.
Cause: When the frontend saves settings to Supabase, it builds a snapshot using buildAgentSettingsSnapshotFromState() which did not include availableToolDetails. This overwrote the entire settings column, wiping the tool definitions the ToolRunner had set.
Fix: Preserve availableToolDetails and availableTools in the settings snapshot so they survive the round-trip.

Duplicate agent profiles

Symptom: Inconsistent behavior — sometimes tools work, sometimes they don’t.
Cause: Multiple ToolRunner instances each created their own robot-demo profile with different toolCallbackUrl ports. The platform randomly picked one, often pointing to a dead process.
Fix: Ensure only one ToolRunner instance runs at a time. Delete stale duplicate profiles.

Phantom control_robot tool

Symptom: AI calls control_robot instead of specific tools like move_forward.
Cause: The base ToolRunner._all_tool_details() always prepends self.tool_name as an action tool. With its vague description “Control the robot using the available tools”, the AI preferred it over the specific tools.
Fix: Override _all_tool_details() in RobotToolsRunner to only expose the 16 real tools.

G1 engine had duplicate system instruction

Symptom: G1 engine behavior drifts from other engines after system instruction changes.
Cause: g1-engine.ts had its own copy of buildSystemInstruction() instead of using the shared version from _engine-common.ts.
Fix: Removed the duplicate and imported the shared function. All four engines now use the same system instruction.

Cooldown window columns crash

Symptom: All API calls return FUNCTION_INVOCATION_FAILED.
Cause: The 5-hour cooldown window feature wrote cooldown_window_start and cooldown_window_usage columns unconditionally, but they didn’t exist in the Supabase usage_metrics table.
Fix: Guarded with hasMetricsColumn() check. Created a Supabase migration to add the missing columns.

Tool name normalization

Symptom: “move forward” returns “Unknown tool: move forward”.
Cause: The ToolRunner’s HTTP server matched tool names with exact string equality. A space in the name didn’t match the underscore in move_forward.
Fix: Added .replace(" ", "_") normalization in the tool server’s POST handler.

9. Running the Tests

All tests require the simulation and ToolRunner to be running (Terminals 1 and 2 from the Quick Start).

test_all_engines.py — Single-Turn Across All Engines

Sends 20 prompts per engine through /api/chat with full tool definitions — the exact pipeline the browser UI uses. Validates that each engine correctly selects tools and extracts parameters.

OMNI_KEY="olink_..." python -m omnilink.examples.robot_demo.test_all_engines

test_explore.py — Long-Horizon Exploration

Multi-turn tool chain with tool results fed back to the AI, running up to 50 rounds per task. Tests the full tool-result loop with sustained multi-step planning.

OMNI_KEY="olink_..." python src/omnilink/examples/robot_demo/test_explore.py g3-engine

simulate_ui.py — UI Pipeline Simulation

Replicates the exact path the frontend takes: wraps user text in messages, includes conversation history, runs through the prompt pipeline, and sends to /api/chat — identical to what the browser does, but called directly from Python.

OMNI_KEY="olink_..." python -m omnilink.examples.robot_demo.simulate_ui

10. Token Usage & Cost

Extended benchmarks measuring token consumption and cost over sustained sessions using the G3 engine (Grok 4.1 Fast Reasoning) through the full UI pipeline (/api/chat). The simulation resets between tasks so battery is always fresh.

Token Consumption

Rates

Estimated Cost (Grok 4.1 Fast: $0.20/1M input, $0.50/1M output)

Projected Costs

Key Observations

• Stable token rate at ~13,000 tokens/min regardless of session length, confirming predictable scaling.
• Cache hit rate improves over time (39% at 10 min to 65% at 60 min) as the system instruction and tool definitions get cached by the provider.
• Extremely cost-effective: ~$0.13/hour at Grok 4.1 Fast pricing. A full 24-hour continuous session costs only ~$3.23.
• Completion tokens are ~3% of total — the AI is concise, spending most tokens on the system instruction + tool definitions in the prompt.
• Each task averages ~11,400 tokens and 3–5 API rounds, making per-task cost under $0.002.
• Zero errors across all three benchmarks (673 total tool calls).

11. Troubleshooting

“Agent says offline” or tools not working

This usually means tool definitions are not reaching the AI. Check that:

• The ToolRunner (run_tools_demo.py) is running and printed the startup banner with the tool server URL.
• The agent profile has allowToolUse: true and availableToolDetails with all 16 tools.
• The toolCallbackUrl in the profile is reachable from your browser (try opening it directly).
• The frontend is sending tool definitions in systemInstructionRequest with each chat request.

Multiple instances running

If you get port conflicts or stale state, check for lingering Python processes:

# Windows
wmic process where "CommandLine like '%robot_sim%'" get ProcessId,CommandLine

# Linux / macOS
ps aux | grep robot_sim

Kill any stale processes and restart cleanly.

“Unknown tool” errors

Tool names use underscores, not spaces. The correct names are move_forward, go_to, pick_up, etc. If the AI is generating tool names with spaces or hyphens, the tool definitions may not be reaching the provider — check the profile’s availableToolDetails.

Battery depleted — game over

When the robot’s battery reaches 0%, the game enters a “game over” state. Movement tools will return error messages. The AI should recognize this and stop issuing movement commands. To reset, restart the simulation (robot_sim.py).

Tools return errors but the chat works

This means the browser cannot reach the toolCallbackUrl. Check that:

• The simulation (robot_sim.py) is still running on port 5050.
• The ToolRunner’s tool server is still running (check its terminal for errors).
• If accessing from a different device, the toolCallbackUrl uses the correct IP (not 127.0.0.1).
• Your firewall allows inbound connections on the ToolRunner’s port.

Pygame window unresponsive

The HTTP server runs in a background thread. If the Pygame window loses focus, click it to restore. Alternatively, the ToolRunner starts a headless simulation automatically if no display is detected.

OMNI_KEY not recognized

On Windows PowerShell: $env:OMNI_KEY = "olink_...". On Unix shells: export OMNI_KEY="olink_..." or prefix the command as shown in the Quick Start.

12. File Structure

All files live under omnilink-lib/src/omnilink/examples/robot_demo/.