Tool calling is the benchmark that matters
I benchmarked ten open models for agent work on my own GPU server and a laptop. Tokens per second got all the attention. The result that decided everything was whether the agent loop survived its second turn.
I run my own inference server at home: an Ubuntu box with an Intel Arc Pro B70, a workstation GPU with 32 GB of memory, served through llama.cpp and later vLLM, reachable over a Cloudflare tunnel. It is the backend for my agent experiments. Over the summer I benchmarked ten open models on it and on a laptop, to pick the one my agents should use.
I expected the decision to come down to speed. It did not.
The throughput numbers
Figure 1. Decode speed on one B70, llama.cpp, mostly Q4_K_M quantisation. Bars show the measured range.
| Model | Type | Tokens per second | Verdict |
|---|---|---|---|
| Nex-N2-mini | MoE, 262K context | 50 to 70 | Chosen as the agent backend |
| Qwen3.6-35B-A3B | MoE | about 63 | Strong alternative |
| Gemma 4 26B-A4B (QAT) | MoE | 40 to 55 | Evaluated |
| GPT-OSS-20B | MoE | 40 to 50 | Rejected: weak tool use |
| Qwopus3.6-27B Coder | Dense | 25 to 35 | Strong at code, slower as a dense model |
Look at GPT-OSS-20B. Its throughput was perfectly respectable. It was rejected anyway, because in agent workloads it was poor at calling tools correctly. A fast model that calls the wrong tool, or calls the right one with broken arguments, just fails faster.
The bug that taught me the most
Later I moved serving from llama.cpp to vLLM on Intel’s XPU backend, with FP8 KV cache, prefix caching and speculative decoding using the model’s own multi-token prediction heads. The speed-up was large.
Figure 2. Same model family, Qwen3.8-27B. On the B70, moving from llama.cpp to tuned vLLM took real-world decode from about 10 to 11 tokens per second to 63 to 77. On the laptop, a speculative-decoding runtime (MTPLX) tripled throughput over plain mlx-lm.
There was a catch, and it is the most useful thing I learned all summer. With the wrong configuration, every agent session ends after exactly one turn. No error, no crash.
The cause was a single flag. vLLM needs to know how to parse a model’s tool calls. With the parser I had configured, the tool calls came back as plain text and were never recognised as calls. The agent saw an answer with no tool call, decided it was finished, and stopped. The right parser for this model family fixed it instantly.
| Setting | Effect |
|---|---|
| Wrong tool-call parser | Tool calls returned as text; agent loop silently ends after one turn |
| Correct parser and auto tool choice | Multi-turn agent loops work |
| Speculative decoding on, sequence reaches the maximum length | Engine crash: a known defect, worked around by raising the context to 262,144 tokens |
None of these show up in a tokens-per-second benchmark. All of them decide whether an agent works.
How I test models for agents now
| Test | What it catches |
|---|---|
| Run a real multi-turn agent task end to end, not a single prompt | Silent loop exits, parser problems |
| Count correct tool calls out of all tool calls | Models like the one I rejected |
| Push a session to the context limit on purpose | Crashes at the boundary |
| Measure throughput last | It only matters once everything above passes |
The hardware question
I also tested whether to move local inference onto a new laptop. On the laptop, a 27B model reached about 41.5 tokens per second decode and 410 prefill with the right runtime, against 13.9 with the default one. The conclusion I came to: 64 GB of unified memory is the practical minimum for capable local models, and for my usage a hybrid (a home GPU server for private and bulk work, frontier APIs for the hardest tasks) beats buying the biggest machine.
That is also how I would advise an enterprise to think about open models. Do not choose by leaderboard speed. Choose by whether your agents complete real tasks, safely, at the end of a long session. Then optimise speed.