Evaluation
- Tool calling is the benchmark that mattersI benchmarked ten open models for agent work on my own GPU server and a laptop. Tokens per second got all the attention. The result that decided everything was whether the agent loop survived its second turn.
- Small models don’t know EARS. So I’m testing before I train.I tried open models up to 12B on a simple job: rewrite an existing requirement into EARS notation without inventing anything. None did it reliably. Here is the evaluation I built before training a 350M helper model.