From Model Benchmarks to Model Science

It’s been a while since the last post. Many exciting things happening at the same time in different fronts: new assignment, new lab, better toys, but also far less time for playing. Despite that, we’ve continued to refine our Zero-Trust + Local AI architecture with better model benchmarking and we wanted to share the results with you.

How a broken LiteLLM experiment became a better way to qualify local agents

In previous posts we started with a hardware problem related to the use of local agents on a real workstation, not on a conveniently rented abstraction where VRAM (and information privacy)is someone else’s problem (ehem… hyper-scalers.)

The context of all these efforts is Zero Trust AI sovereignty, where our first benchmark asked what an advanced AI user could get from consumer hardware. Once we selected and setup our local AI infrastructure, the next phase was building a hybrid routing to make better use of the different model’s capabilities, but we discovered that bad configuration can mask the gateway’s apparent success, letting the fallback illusion use a big/cloud model (in this case it was Gemini) do the work while the local model wrongly receives the credit.

After smart routing was solved (and btw we did this before Switchyard was a thing) we started optimizing the dynamic model selection. The Benchmark Paradox followed, that discovery with a harsher gauntlet where we deliberately broke models with strict formatting, tool schemas, context switches, and architectural tasks gave us a simple and slightly rude lesson: a model that is eloquent in a chat window may still be a liability inside an agent loop.

This post is the next logical step in that sequence of experiments: Once the routing path was isolated and the models were being tested against agentic work, we still had a measurement problem. In our quest to discover what models worked well with Agentic Frameworks we had build a better test, but not yet a sufficiently honest instrument: that version of the testbench for local-models only simulated what an agent would do, we didn’t use an actual agentic framework.

A model can return HTTP 200 and still be useless to an agent. It can emit something that looks like a tool call without actually calling anything. A proxy (Litellm in this case) can silently drop the tools array. A configuration can work through one provider adapter and fail through another. A test can pass because a file appeared somewhere on disk, even though the model never used the file tool that was supposed to create it, and even more worrisome the benchmark can look healthy while in reality the agentic framework is delegating work to an agent that is quietly holding the digital equivalent of a cardboard wrench.

This is a continuation of the work described in The Benchmark Paradox, Engineering Past the VRAM Iron Curtain, and The Hallucination Sandbox.

The earlier articles were about the physical limits of local inference: VRAM, routing, coexistence, and the uncomfortable gap between a model that can answer and a model that can operate. This round changed the question.

Instead of asking, “Which model passed?” I started asking:

What exactly does a passing result prove, and can another person reproduce that claim after the machinery changes (you know: gitops)?

That question turned a model sweep into a full qualification protocol.

A bit of the lab’s context

Before digging into the specifics this new experiment, lets recall the topology over which we operate. We used an opinionated architecture to achieve not only AI sovereignty but also zero trust. The Ollama model service and its proxy (LiteLLM) sit behind a router that isolates the infrastructure from the Experimental Agentic Frameworks (Open Claw and Hermes). There are no exposed or forwarded ports but to the API gateway (APISix) that routes the calls.

We chose Ollama server because it allows us to load multiple small LLMs (e.g. a reasoning and an embedding model, when VRAM allows it) and also dynamically switch between the available model roster within tolerable response times. This can take a toll in you GPU’s VRAM so we carefully tuned model’s persistence and agentic profile configurations to reduce hammering our RTX 3090Ti by constantly loading and unloading models.

We chose LiteLLM because it allows to be configured through a yaml file (which aligns with our gitops approach for architectural reproducibility) and also has many useful features like multi-provider support (local and cloud), rich logging, token tracking, user and key management, model alias an fallback definitions, and more.

APISix, aside from it’s already stated purpose, comes with many built-in plugins for rate-limiting, authentication, proxy-rewrites and many more that help isolating and securing the infrastructure behind it. This means that even if a model runs amok, we have several mechanisms to avoid it blocking or crashing the local AI infrastructure, including a remotely side-channel emergency kill switch that severs the Agent’s ability to connect to the model’s provider.

The first version was too easy to fool

The original experiment to benchmark the local models was built around a sensible-looking matrix: discover the models from Ollama, test several LiteLLM provider routes, exercise single-tool and multi-tool calls, send tool results back, and record the outcome.

From our early test we know that the model provider configured in LiteLLM mattered. The same model could behave differently through ollama, ollama_chat, and an OpenAI-compatible route. In one particularly useful failure, the model was capable of producing native tool calls directly, but LiteLLM’s plain ollama adapter dropped them. So switching the model’s configuration to ollama_chat restored the calls. The model had not changed but its wiring had, and that was one of the advantages of having the model’s configuration in a file: we could programmatically patch it, test the results, roll it back and even version it.

That was already a useful lesson: Switching model names is a good exercise but the real experiment is the model plus its target, provider, profile, stream mode, and attempt.

So, our the first version of the matrix benchmark treated each combination as a row, which was right. but the evidence model was still too optimistic. The initial form of our tests on tool calls asked Hermes to inspect the date and disk usage of its host computer (to which it case root access), write the results to a file, and report back. Then the test runner checked the log and the resulting artifact. If the log contained phrases that suggested a command had run and the file looked plausible, the row could pass.

However, we discovered that that only checking the logs was a dangerous shortcut we took: A language model can write:

<function=terminal>

without producing a structured tool call. It can describe a file it intended to write. It can write a file through one path while the harness checks another. It can produce an artifact that looks real because the surrounding environment supplied enough context for the model to imitate reality, but without actually producing the intended outcome.

What we needed was not logs but actual evidence: the output may be persuasive, but proper evidence is a hard fact.

The result that forced the redesign

In one instance running test on the qwen3-coder model, its results exposed the problem cleanly: an earlier run had recorded a mixed result, including a passing streaming configuration but when the new (and stricter) “receipt-based” runner was applied, the model produced pseudo-tool syntax instead of executing the tools.

That was not a minor parser bug, it changed the whole claim of the resulting row in our results matrix! A result saying “the model used tools through Hermes” is much stronger than a result saying “the model produced text that looked like a tool plan and an artifact was found somewhere”.

The correction was unpleasant because it invalidated a more flattering result that had proven ambiguous in real-life use cases. It was also the point of the work we were doing. We found that we had a qualification system producing bold claims that were nor backed by evidence.

The same thing happened with Granite. An earlier run suggested all three tested routes had passed. Under the receipt-backed procedure, the model failed to provide reliable tool execution evidence. The new test made the failure visible.

This was the difference between debugging the model’s configuration and debugging the measurement we were using for baseline.

The before-and-after result is therefore not a prettier leader board, it is a change in what a “green cell” in the benchmark matrix means. Before, a passing row could mean that the request completed and the surrounding output looked plausible. With this new test, it means that the exact route produced structured execution evidence, a matching tool result, and a verifiable artifact. The stricter result can contain fewer passes and still be more valuable.

A benchmark row is not a scientific claim

Our previous attempts naively assumed that simulating a tool call was enough, so (as we mentioned) this time we evolved our single “tool call” test to a three separate verdict process:

• The first is the provider-row verdict:

target × model × provider × profile × stream_mode × attempt

This says whether that exact configuration passed.

• The second is the mechanism verdict. Did the qualification procedure produce trustworthy evidence? If one provider fails but another provider for the same model produces a valid receipt and artifact, the provider failure remains important, but it does not mean the entire mechanism failed.

• The third is the best-observed configuration. For each model, choose one verified configuration from the experiment results. That is useful for producing a reviewable LiteLLM handoff. It is not a universal statement that the model is best, or that the chosen route will remain best after a component upgrade.

Those 3 claims are related, but they are not interchangeable. The earlier versions of our matrix were letting them blur together: a model passed, a provider passed, the benchmark passed, therefore the configuration was optimal, but in practice the use of some of the “passed” models was sub-optimal and it ruined real work (e.g. when delegated tasks in Kanban boards chose any of those misbehaving models and just simulated the tool calls, effectively blocking other agents that depended on that work).

Note that this test is designed to only prove whether a model can configured through LiteLLM for Agentic tool use in Hermes or Open Claw, that’s it. We are not claiming that the results imply that the model will correctly use the tool or provide the intended outcome for every possible request. For this test we are only interested in the tool use capabilities of the model, not its performance.

What counts as a real Hermes tool receipt

To implement the 3 verdict process we had to redesign the Tool Use test. We named it Gate 5 runner and we decider to use Hermes’ top-level one-shot mode:

hermes -z <task> --usage-file <path>

Note that the Hermes chat subcommand does not accept --usage-file (that small detail derailed our efforts for a while because the two command forms look close enough to encourage confident guessing :S).

With one-shot mode, the usage report supplies a session ID, the Gate 5 runner then inspects the isolated Hermes state.db, where the actual message history contains structured fields for assistant tool calls and tool results.

We designed the test so that a valid receipt requires:

1. A completed Hermes usage report with a session ID.
2. An assistant message containing structured tool_calls.
3. Matching role=tool messages.
4. Matching tool_call_id values.
5. The expected tool names.
6. Non-empty result content.
7. A valid artifact written inside the experiment sandbox.
8. A structured write_file call whose arguments point to the expected result file.

After some trial an error we enforced that model prose does not count, XML-like pseudo-calls do not count, a line in stdout saying “I wrote the file” does not count.

So, the script for test runner that serves as the receipt extractor is intentionally short and boring: It reads rows, joins calls to results, checks names and IDs, the contents of the resulting file, and reports what is missing. Short and boring is good here because the agent get be creative but the evidence should not be.

Note that because this is such a constrained test, we are assuming that tested model or (Agentic framework configured) will not tamper the state.db or perform an intricate evidence fabrication.

The sandbox became part of the experiment

The artifact problem required another change. Initially we were asking Agent configured with a local (and therefore small) model to reproduce a long absolute path containing timestamps, provider names, model tags, etc. It was ultimately unnecessary and error-prone.

We setup the lab to have a test Hermes Agent pointing to an IP-fixed but ephemeral LiteLLM instance that was configured via a YAML file, and we had a “Production” Hermes Agent running the battery-included of tests against 28 local models served by our Productive Ollama Server but accessible only through the LiteLLM ephemeral Proxy. The production Agent was specifically told not to change any code from the locally accessible LiteLLM ephemeral instance or the test Hermes Agent, the only changes allowed were on the LiteLLM’s config YAML file.

We also designed the Gate 5 runner to own the experiment directory and give to intentionally give the models being tested with a short filename such as:

result.md

and a predictable sandbox scaffolding:

runs/gate5/<experiment-id>/
  hermes-home/
  result.md
  usage.json
  hermes.log
  receipt.json

Hermes was configured to work in that directory and the Gateway 5 runner verifies both the structured write_file arguments and the resulting file location.

This distinction matters because the model may report that it wrote /home/rod/agentic_test.txt but if the experiment expected result.md inside its sandbox, that is not a successful artifact. It is evidence that the model and harness “disagreed” about the workspace and that disagreement is itself a result.

The manifest became the record, not the decoration

In the previous version of our test bench, a separate results file was convenient but we found that if often did not matched the manifest. Then someone had to decide which file was to be used as the source of truth to set the Productive LiteLLM instance configuration (usually by opening both and trusting whichever one was edited later).

The project now merges Gate 5 results back into the canonical manifest using the full experiment identity. The merge is append-aware and refuses to attach a result to an unknown row.

That gives each row a durable record of:

• automated gate status;
• Hermes Gate 5 status;
• receipt and session ID;
• artifact path and validity;
• elapsed time;
• provider and stream claim boundary;
• component fingerprint;
• promotion eligibility.

The manifest is not just a table for a report anymore, it became the actual experiment ledger.

When a new model is added, incremental qualification appends only missing tuples, so subsequent runs do not quietly rerun the whole catalog, overwrite old attempts, or pretend that a new component version produced the old evidence. This is where the system starts to become useful beyond this particular batch of models:

• A new model should be a new set of rows
• An updated Ollama, LiteLLM, or Hermes should be a new qualification epoch

This is important because different versions of the underlying infrastructure may patch bugs or ship new features that change the behavior of a model under a specific configuration.

Additionally, the old results remain historical evidence.

Components move, so evidence needs fingerprints

To align to our gitops approach, the ephemeral LiteLLM instance was containerized and we originally used:

ghcr.io/berriai/litellm:main-latest

That was convenient for experiments but turned out terrible for reproducibility. A tag that points somewhere new will eventually point to something different to what the current setup is (even if the configuration YAML did not change).

We updated the harness to pin the LiteLLM image by digest and now it records:

• LiteLLM image tag and digest;
• image ID;
• Hermes version;
• Python version;
• platform information;
• runner and protocol versions;
• elapsed time.

With this changes we can evaluate an upgrade in any of the components as a new epoch. If a model changes from pass to fail, the first question is not “which prompt did we accidentally change?” but “based on the last run, what part of the topology or configuration changed?”

That is a much shorter path to the truth (and a whole lot less of debugging hours indeed).

Stream mode needed a claim boundary

During our research to find the optimal LiteLLM configuration for a model, we learned that LiteLLM can explicitly send:

{"stream": true}

or:

{"stream": false}

However, Hermes’ one-shot CLI does not expose a reliable per-run stream switch. So the results now have to separate claims so that stream and non-stream rows in the automated proxy gate are labeled:

proxy-agentic-compatibility

They prove behavior through the OpenAI-compatible proxy with the tested request mode which makes the result confidently state what was actually tested.

Model promotion is now a hand-off

The generated configuration uses semantic aliases rather than forcing every downstream consumer to know the physical model route.

For example:

local-agent-vision-ollama-openai-ornith-35b

The alias points to the tested provider and model, while the generated configuration keeps endpoint and credential values as environment references. Models with no verified configuration are omitted from the hand-off and listed in a rejection report.

That rejection report matters because in a YAML file absence can be ambiguous. A model may be missing because it failed, because it was never tested, because its row was filtered out, or because someone forgot to add it. A separate report turns omission into an explicit decision.

The result is still not yet an automatic production deployment but it is a reviewable proposal with provenance that someone can check (aliases, the environment mapping, the component epoch, the intended role) before deciding if it can be promoted from staging with the potential of being it being fully automated further down the line (e.g. given the LiteLLM’s config template, an LLM can fill it out with the new “best-observed” authoritative results from the last run).

The Before vs the After: the uncomfortable upgrade

The difference between our previous benchmark and the new one is easier to see as a table than as a sermon.

Before	After
HTTP status and response shape were strong signals	HTTP status is transport evidence only
Logs and model prose could suggest a tool had run	Hermes state.db must show structured tool calls and matching results
A plausible file could satisfy the artifact check	write_file arguments and the sandbox path must agree
One result file could carry several interpretations	Each row has immutable experiment identity
A mutable main-latest image was acceptable	LiteLLM is pinned by digest and components are fingerprinted
A provider failure could blur into a model verdict	Provider-row, mechanism, and best-observed verdicts are separate
Adding a model tended toward a broad rerun	Incremental qualification appends only missing tuples
Stream labels implied more than the runner could prove	Stream claims are explicitly scoped to proxy compatibility

The old system was not useless, it helped us found real provider differences and exposed genuine routing failures but it was simply too willing to promote plausible output into factual evidence. The new system is less generous and therefore more useful.

What the current epoch taught us

The fresh receipt-authoritative Gate 5 epoch ran 37 exact candidate rows with the pinned LiteLLM image and dedicated staging credentials.

It recorded 20 verified rows and 17 failed rows, with 11 models retaining at least one verified configuration. Those counts are less important than the fact that they differ from the earlier run and thus the new matrix became authoritative (even if it was worse). The point was not to improve the percentage of green cells but to make a green cell mean something defensible.

That distinction changed the interpretation of “usefullnes” of several models. Some failures were provider-specific, some were stream-specific. But at least now we have a reliable source of truth from which to select each model’s best-observed configuration and tested capabilities to map to a list of intended roles (e.g. MoE, Kanban Profiles, LLM Councils, etc.).

The approach has already proven useful by saving about 2 hours of testing for setting up the new Qwen3.8:27b. We ran the test and had the model running in production for bot Open Claw and Hermes agentic frameworks less than 5 minutes. We joked about that this time it took longer to update Ollama to the latest version than what it took to have the new Qwen model running.

What this enables next

The real output of this project is not the current YAML file. Models age quickly, providers change behavior, and local hardware has the emotional stability of a cat near a glass of water.

What we achieved is not a “stable/reliable” model configuration catalog but a systematic approach for change control.

With the new process, adding a model can mean:

1. Download the new model to the local Ollama server.
2. Discover the model from the live catalog.
3. Append only missing experiment tuples.
4. Run isolated provider comparisons.
5. Perform selective tests (with a nice test resume feature).
6. Run the actual Hermes task with a structured receipt.
7. Merge the result into the canonical manifest.
8. Select a best-observed route for that model.
9. Review the generated hand-off without touching production.
10. Run the protocol manually or delegate the task to an AI Agent
11. Decide in less than 5 minutes whether the model works or not

When Ollama changes, repeat the qualification epoch. When LiteLLM changes, pin the new digest and compare. When Hermes changes, record the new version and recheck the receipt path. If the result moves, the project has a way to ask why.

That is the difference between our first experiment as just benchmarks and the new process as scientific instrument. The instrument does not promise that every reading will be flattering but it ensure that the reading of the results cant be tied to a known relationship that factually reflect what happened for each particular run.

The benchmark is still useful. It just needs less authority

Benchmarks are good at producing numbers for model comparison but they are bad at explaining what the numbers mean unless the experiment has been designed with a particular problem in mind. That’s what the many popular benchmarks for ranking models do, they frame the comparison over a strict an scoped domain.

But we are not testing models for a restricted set of tasks, we want to know if they are fit for our very opinionated local AI architecture, and in what capacity. A model can be fast, capable, and still fail the exact provider route an agent needs. A provider can fail while another route works. A stream result can be valid for proxy compatibility without proving that Hermes itself used that stream mode. A file can exist without the file tool being called. A run can finish successfully while producing no usable evidence.

That is why our current approach is prescriptive:

• define the experiment identity before running it (the versions of all components involved);
• isolate the topology (create a dynamic ephemeral containerized matching architecture);
• separate model, provider, and mechanism claims (test all the combinations for the relevant parameters patching the LiteLLM configuration file);
• preserve failed rows (models not passing);
• require structured execution receipts (factual evidence);
• fingerprint changing components (in case any component of the architecture changed);
• label stream boundaries honestly (due to its non-strict enforcement over the whole request chain) ;
• generate a reviewable hand-off instead of silently deploying the winner (the resulting manifest file).

This is more work than sending a prompt to thirty models and counting check-marks but it is also far less work than debugging failing production agents whose benchmark passed for reasons nobody can reconstruct.

The current (and also upcoming new) models are and will still be tested, but now the tests are finally telling us the truth, and so far less than 20% of our local model roster has true agentic capabilities for the tested agentic frameworks.

It’s been quite a journey to get to this point, and now, we are stepping up our game with our brand new DGX Spark with an already running DeepSeek-v4-flash model (thanks to Antirez and Entrpi). Stay tuned!