Figures from the eval runs of 21 and 28 October 2026: the results page and the public eval repository.
We measured it: 300 questions a freight desk actually asks, scored deterministically, run on four frontier models alone and with the FreightUtils tools, and on one of them with its provider's web search instead of the tools. Everything is public: the questions, the answers, the scoring rule, the spend.
The questions
The questions fall in six families: ADR Table A values (class, classification code, packing group, labels, limited quantity, tunnel code), 1.1.3.6 small-load totals, ADR 2025 → ADR 2027 changes, HS 2022 hierarchy, freight arithmetic (chargeable weight, CBM, loading metres), and identifiers (UN/LOCODE places, AWB prefixes, ULD specifications). 30 of the 300 ask for ADR 2027 values — the amendment accepted on 9 October 2026, applying from 1 January 2027 — which no tool served when the runs were made.
Scoring is a string rule, not a judge model: the last line of the answer, or the first number on it, against the expected value. An answer can be right, wrong, or "unknown". Wrong answers are the ones to watch, because an agent acts on them.
What the models do alone
Outside ADR 2027, the four models answered 90–95% of the questions correctly from their own weights. Three of them said "unknown" on most of the rest; all of them were confidently wrong on 1–5%.
The wrong answers cluster where you would least want them: tunnel restriction codes, 1.1.3.6 totals, limited-quantity limits, UN/LOCODE places and AWB prefixes. Every confident-wrong answer was checked against the primary source — the UNECE Table A workbook, UN/LOCODE, the carriers' own pages — and every one was a model error, not a question error.
On ADR 2027 the models answered 53% at most, and up to 30% of their answers were confidently wrong: they did not know what the amendment changed.
With the tools
Outside ADR 2027, two of the three models run with the tools answered 270 of 270 and the other 269 of 270 (one label code): 809 of 810 in all. The fourth model's tools run did not fit the budget. Confident-wrong: 0–0.4%. Unknown: 0%.
On ADR 2027 the tools made things worse for the honest reason: they served ADR 2025, and a model that trusts its tool answers "unknown" instead of guessing. The model that had answered ADR 2027 alone went to 1 of 30 right, 29 "unknown", 0 wrong. That is the behaviour you want from an agent, and it was the gap we closed next.
ADR 2027 on the tools: the v0.4 run
On 30 October 2026 the tools began to serve ADR 2027: the five tools that read ADR data take edition (ADR 2025 or ADR 2027) and as_of (a date), and without either they answer from the edition that applies on the day — ADR 2025 until 1 January 2027, ADR 2027 from then, with a notice while ADR 2025 may still be applied. An answer on ADR 2027 also carries the transitional measures that keep an ADR 2025 value usable, such as the old 1.1.3.6 rule for the nine "note a" entries, beside the verdict rather than in place of it.
We asked the same 30 ADR 2027 questions again, with the tools. Three models answered 30 of 30, where in the earlier run no model with the tools had answered more than one; the fourth model was not run (its account was out of credit). On all 300 questions, the model we re-ran answered 300 of 300.
The same run measured what the code meanings cost. adr_lookup can return what each Table A code means beside the row (include_codes: none, compact — the default — or full). On the Table A and 1.1.3.6 questions every mode answered every question right, and the cost per correct answer was 0.0398 USD with none, 0.0578 with compact and 0.1131 with full: 1.45 and 2.84 times the cost of none. Those questions ask for a value, not what a code means, so there the meanings added tokens and nothing else. If your agent reads the value and nothing else, send include_codes=none; leave the default where it has to explain a code.
Questions that need what a code means: the v0.5 run
On 30 October 2026 we added 40 questions whose answer is fixed by what a Table A code means — a packing instruction's limit, a special provision's threshold, the 1.1.3.6 maximum of a transport category, a tank code's calculation pressure — and asked them again in each answer mode. With the tools the model answered 38 of 40 in every mode, and every answer it gave was right: the two misses were questions the provider's API blocked before the model replied, as were the six misses alone, where it answered 34. The cost per correct answer was 0.0835 USD with none, 0.0853 with compact and 0.1705 with full: 1.02 and 2.04 times none. Without the meanings in the answer the model fetched them itself — 29 extra lookups against 11 with compact — so on questions like these none saves almost nothing. Keep the default when your agent has to say what a code means; full doubles the cost and changed no answer.
The cost: a tool call per question, and the tool manifest itself. With 24 tools described in one manifest (about 33,000 tokens, written to the cache on the first question), the tools condition cost between 5 and 60 times the alone condition per answer, depending on the model. Scoped manifests — a dangerous-goods agent loads just the dangerous-goods tools — are the fix, and they now exist: /api/mcp/dg, /api/mcp/customs and /api/mcp/ops on the hosted server, and --scope in the npm package, each a filter on the full list: a tool in a scope answers exactly as it does in the full list, under the same key and limits.
With web search instead of the tools
On one model we replaced the tools with the provider's web search (up to five searches per question, no FreightUtils tools). Outside ADR 2027 it answered 268 of 270. It never said "unknown". It was confidently wrong on 2 of 270 — one of them the wrong label code for a lithium-battery entry, the same mistake another model made in a different run.
Outside ADR 2027 it cost 1.6× the tools condition per answer and took 2.6× as long, because search results are billed as input tokens and the model reads them through code.
On ADR 2027 search found the amendment documents and answered 77%, with 23% confidently wrong. Four of those wrong answers carried the right value with a source line after it, which the scoring rule does not accept; the other three misapplied a change it had read.
Two more things the search run showed. First, the pages it read were mostly third-party reference sites and Wikipedia, not the regulator's text. Second, it read freightutils.com on 3% of the answers — which told us our pages were not written for the way a search-equipped model picks what to open. We changed that the same week: every UN-number page now leads with the Table A values in ADR's own vocabulary and prints all twenty columns, and the API serves them too.
What this means if you are building an agent
- Search closes most of the accuracy gap on lookups, and opens a different one. 99% right is impressive until you notice the failure mode: never "unknown", always confident. A reference tool that answers "unknown" when it does not serve something is a safety property, not a weakness.
- Editions are the real product. A tool is exactly as current as its edition. The ADR 2027 family is where a tool with the amendment will beat search outright, because search got 23% of the ADR 2027 answers wrong and did not know it. Ask any reference vendor two questions: which edition, and what happens on the day the next one applies.
- Price the whole call, not the API. Our API call costs nothing; the model's tokens for the manifest and the result are the cost. A manifest of 33,000 tokens is expensive; scoped manifests and batch inputs are what make a reference tool cheaper than a search.
- Trust the receipt, not the sentence. Every FreightUtils answer carries its source, edition and checked date. On the REST API, asked for envelope
1.2, it also carries a status (ok / partial / not_found / unsupported / ambiguous) and the inputs it was missing; the MCP tools answer with envelope1.2too, refusals included. Build your agent to read those fields, not the prose.
Reproduce it
The eval repository holds the question set, the harness, the scoring rule, every answer and the spend. The results page on freightutils.com renders from a pinned copy of the results and nothing else, and a lint proves every figure on it matches. Run it on your own model; if you get different numbers, we want to know.
Written by the person who builds and runs FreightUtils — a UK transport planner by day. The tools are free to use at the volumes an evaluation needs.