In our last iteration of the OpenClaw lab, we built a hybrid routing engine. The architecture was sound: use the API Gateway and LiteLLM to dynamically toggle between cloud-tier reasoning models and local, VRAM-constrained models. We had the physical infrastructure, the isolated sandboxes, and the VaultBridge protecting our secrets.
But a router is only as intelligent as its routing table.
If we were going to offload tasks to an LLM, we needed to know exactly what each model was capable of. The obvious solution was to check the public leaderboards (look at MMLU scores, check the LMSYS Chatbot Arena, and pick the winners).
We tried that. It failed catastrophically.
Public benchmarks test for conversational fluency, trivia recall, and standardized test-taking. They favor models with high saturation on common training datasets. But we don’t need a conversationalist. We need an agent. We need a model that can read a log file, call a tool, parse the JSON response, and formulate a next step without hallucinating an API endpoint.
The last few weeks in the lab have felt less like software development and more like conducting an autopsy on a live patient. We had to stop guessing. To move past the marketing fluff and leaderboard padding, we had to build our own protocol.
So we had to broke our own models to find the truth.
The Core Failure Mode: “The Agency Conflict”
When you use an LLM in a chat interface, the model has one job: predict the next token to satisfy the user’s prompt.
When you put an LLM in an agentic loop (like OpenClaw), the environment is violently different. The model is bombarded with system instructions, strict JSON tool schemas, conversation history, and hidden system prompts.
What we found wasn’t just a slight performance delta between models. We discovered a recurring, fundamental failure mode we call the Agency Conflict. When forced to operate in these loops, models begin fighting their own training. They collapse under the weight of their own prompts. They hallucinate tool schemas, forget their constraints, and spiral into execution loops that burn through tokens until they hit a timeout.
Intelligence in an LLM is a fragile state. The moment you introduce an agentic loop, you are forcing a probabilistic text generator to act like a deterministic state machine. Most of them cannot handle the friction.
The Five-Stage Gauntlet
To quantify this, we designed a custom 5-stage benchmark tailored specifically for “Research and Software Development” tasks. We aren’t measuring peak tokens-per-second; we are measuring resilience.
We needed to know how these models behave when they are pushed, constrained, and forced to synthesize logic under VRAM-starved conditions.
Stage 1: Constraint Compliance & Recall
Before a model can reason, it must follow instructions. We split this foundational test into two parts:
- 1A (Formatting): Can the model follow an output format without breaking the schema? We demanded a strict
READYsignal. No conversational fluff, no “Here is the status you requested.” Just the exact string. If a model cannot adhere to a basic text constraint, it cannot be trusted to generate valid JSON for a tool call. - 1B (Recall): Does it remember the system state? We inject a variable early in the context window and ask the model to retrieve it after several conversational turns. If the KV cache fragments or the model loses context, the agent will inevitably forget the overarching goal of its current task.
Stage 2: Logical Sequencing
Can it perform multi-step reasoning without “phantom spikes”?
We asked models to map out dependencies for a microservices testing strategy. We weren’t just looking for the right answer; we monitored the VRAM. Some models would suddenly spike in memory consumption, attempting to load massive amounts of irrelevant contextual weights to solve a simple logic puzzle. A model that consumes 22GB of VRAM to write a basic if/else statement is wildly inefficient for a local stack.
Stage 3: Tool Execution & Schema Adherence
This is where the Agency Conflict hits hardest. We introduce a complex tool schema (e.g., querying a Vector Database for test embeddings). The model must formulate the exact JSON required to trigger the tool. Many models simply ignore the provided schema and invent their own parameters based on what they saw in their training data.
Stage 4: System Resilience (The Context Switch)
How does the model handle a simulated failure? We forced the environment to return an error code to the model’s tool call.
A reliable agent reads the error, adjusts its parameters, and tries again. A fragile model panics. It either repeats the exact same failing tool call in an infinite loop, or it hallucinates a “success” message and moves on, completely ignoring the fact that the tool failed.
Stage 5: Architectural Synthesis
The final test. Can the model bridge the gap between abstract requirements and actionable code? We provide a high-level system design and ask the model to generate the exact pipeline configuration required to test it. This tests the model’s ability to act as an Architect rather than just a typist.
The Autopsy: When Models Refuse to Cooperate
We didn’t just get clean data. We got spectacular failures. We didn’t hide them; we mapped them.
Format Drift: Models like the Llama 3 ChatQA 8B suffered from severe conversational bias. When asked for a strict status code, it would return conversational JSON. It wanted to be helpful. In an agentic loop, “helpful” conversational text breaks the parsing logic and crashes the pipeline.
Format Collapse: This was the most alarming discovery. Models like DeepSeek R1 32B and Phi 4 Reasoning 14B suffered catastrophic format collapse. They weren’t just getting the answer wrong; they were outputting raw, unformatted tool-call JSON directly into the chat stream.
Because these reasoning models are trained to “think” before they speak, they couldn’t distinguish between our system prompt instructions and the agentic loop schemas we injected. The model’s internal reasoning engine tangled with our external tool definitions, resulting in a complete failure to execute.
These weren’t just bugs; they were warnings. They proved that agentic reliability is inversely proportional to prompt complexity.
The Current State of the Fleet
After pushing our hardware to the absolute limit, the data forced us to rethink our assumptions. Massive parameter counts do not guarantee agentic competence.
Here is where our routing table stands today. Note that our champion for local execution is not the largest model we tested.
| Model | Status | Path | Score (1A-5) | Observation |
| Gemma 4 31B | Completed | Local | 415 / 500 | Exceptional schema adherence. High VRAM footprint. |
| GLM-4.7 Flash | Completed | Cloud | 415 / 500 | Fast, reliable cloud baseline for logic routing. |
| GPT-OSS 20B | Completed | Local | 370 / 500 | The Sweet Spot. Proves size isn’t the primary driver of reliability. |
| Qwen 2.5 32B | Completed | Local | 335 / 500 | Prone to minor format drift but recovers well. |
(The full, evolving routing table and execution traces are available in our internal master_benchmark_plan.md for our opencalw instance to consume).
The fact that the GPT-OSS 20B sits comfortably in our local stack at 370/500 is a revelation. It fits within the VRAM Iron Curtain of our 24GB RTX 3090 Ti without spilling into system RAM, and it maintains enough logical separation to execute tools without format collapse.
Engineering the Next Iteration
We are still refining the routing engine, but the lesson from the gauntlet is clear: If your model cannot distinguish between your instructions and the system’s own tool-calling schema, it’s not an agent. It’s a liability.
Our next immediate challenge is tackling the Agency Conflict head-on. If the models are collapsing under the weight of complex prompts, the solution isn’t necessarily a bigger model, it’s a cleaner prompt. We are currently working on refining our prompt templates to strip away the noise of schema injection, ensuring the model can focus purely on task execution.
We are going to keep pushing the hardware. If we want to run AI locally, securely, and autonomously, we have to learn to live within the hard physical boundaries of our VRAM.
There is no shortcut here. No magic setting in a .env file that makes a model smarter. Just rigorous, iterative testing, and the willingness to accept that sometimes, the model you thought was an architectural genius is actually just a very good parrot trapped in a very complicated cage.
