On 300 freight reference questions, asked alone, the four models were confidently wrong on 1–5% of the questions outside ADR 2027 (Claude Fable 5.1 (claude-fable-5-1) 2%, GPT-6 Astra (gpt-6-astra) 5%, Gemini 4 Argon (gemini-4-argon) 1%, Grok 4.7 (grok-4.7) 1%). With the FreightUtils tools, the three models that completed the tools condition answered 809 of 810 outside ADR 2027 and were confidently wrong on 0–0.4%.
FreightUtils ran this eval of its own tools. The question set, the harness, the raw per-question results and the scoring are public in the freightutils-eval repository, and anyone can re-run it.
The results
| Model | Condition | n | Accuracy | Unknown | Confident wrong | Mean tool calls | Mean latency | Cost (USD) |
|---|---|---|---|---|---|---|---|---|
| Claude Fable 5.1 | A, alone | 300 / 270 | 86% / 90% | 9% / 8% | 5% / 2% | 0.0 | 2.7 s | 2.12 |
| Claude Fable 5.1 | B, with the tools | 300 / 270 | 90% / 100% | 10% / 0% | 0% / 0% | 1.0 | 3.8 s | 11.51 |
| GPT-6 Astra | A, alone | 300 / 270 | 85% / 94% | 11% / 1% | 4% / 5% | 0.0 | 4.7 s | 1.76 |
| GPT-6 Astra | B, with the tools | 300 / 270 | 90% / 100% | 10% / 0% | 0% / 0% | 1.0 | 7.4 s | 14.05 |
| Gemini 4 Argon | A, alone | 300 / 270 | 84% / 93% | 15% / 5% | 1% / 1% | 0.0 | 9.6 s | 0.24 |
| Gemini 4 Argon | B, with the tools | 300 / 270 | 90% / 99.6% | 10% / 0% | 0.3% / 0.4% | 1.0 | 30.4 s | 14.57 |
| Grok 4.7 | A, alone | 299 / 269 | 85% / 95% | 13% / 4% | 1% / 1% | 0.0 | 30.0 s | 4.40 |
| Grok 4.7 | B, with the tools | not run — xAI credit; needs about USD 30 | ||||||
Each cell gives the figure over all questions, then after the slash excluding the 30 ADR 2027 questions, which no FreightUtils tool serves yet. “Unknown” = the model said it did not know. “Confident wrong” = a definite wrong value. Tool calls, latency and cost are over all questions; cost covers every answer scored. Grok 4.7 condition B was not run: run only if all 300 B answers fit the xAI credit left after A; they did not (estimate USD 29.93, against USD 10 of credit). One answer of Grok 4.7 condition A failed twice with a network error and is not scored, so its n is 299 / 269.
Accuracy by family
- ADR 2025 → 2027 changes (30 questions): Whether a Table A cell changes under the ADR 2027 amendments, and to what.
- 1.1.3.6 exemption totals (30 questions): The ADR 2025 1.1.3.6.4 points total for a load, or whether that total exceeds 1,000.
- ADR Table A fields (70 questions): One cell of an ADR 2025 Table A entry: class, classification code, packing group, labels, limited quantity, tunnel code and others.
- Freight arithmetic (60 questions): Chargeable weight, CBM, loading metres and W/M revenue tonnes from the dimensions and weights in the question.
- HS structure (50 questions): Which HS 2022 chapter or heading a code belongs to. It is a format floor: every model answered all of them correctly alone.
- Identifiers (60 questions): AWB prefixes and airline designators, UN/LOCODE place names, and ULD and container dimensions and weights.
| Model | Condition | ADR 2025 → 2027 changes | 1.1.3.6 exemption totals | ADR Table A fields | Freight arithmetic | HS structure | Identifiers |
|---|---|---|---|---|---|---|---|
| Claude Fable 5.1 | A, alone | 53% | 93% | 81% | 100% | 100% | 80% |
| Claude Fable 5.1 | B, with the tools | 3% | 100% | 100% | 100% | 100% | 100% |
| GPT-6 Astra | A, alone | 0% | 87% | 91% | 100% | 100% | 90% |
| GPT-6 Astra | B, with the tools | 0% | 100% | 100% | 100% | 100% | 100% |
| Gemini 4 Argon | A, alone | 0% | 97% | 93% | 98% | 100% | 82% |
| Gemini 4 Argon | B, with the tools | 0% | 100% | 99% | 100% | 100% | 100% |
| Grok 4.7 | A, alone | 0% | 100% of 29 | 93% | 100% | 100% | 85% |
What the models got wrong
Every disputed value was checked against the primary source, and the served data was right on every one of the 37 confident-wrong answers below. Two questions in the earlier version of the set were the eval’s own faults and were fixed in v0.2:
identifiers-034(scoring artefact; two answers scored wrong that were right): Turin is the English name of the place UNECE lists as Torino; the model is right. In v0.2 the question was kept, and the answer is accepted by the v0.2 place-name scoring rule.identifiers-051(question fault; three answers scored wrong that were right): the served 200 cm is the RKN's overall (contoured) length; its base is the LD3 base, about 156 cm. The model is right. In v0.2 the question was replaced.
| Question | Model / condition | Expected | Answer | Checked against the primary source | Verdict |
|---|---|---|---|---|---|
adr_exemption-0111160 L UN 2934, PG III | GPT-6 Astra / A | 1160 | 3480 | Model error. Transport category 3 (Table A col 15), multiplier 1 (1.1.3.6.4): 1160 points. Checked 2026-10-20. | model error |
adr_exemption-012755 L UN 2053 + 170 L | GPT-6 Astra / A | 925 | 1265 | Model error. Both are in transport category 3 (×1): 755 + 170 = 925. Checked 2026-10-20. | model error |
adr_exemption-024240 L UN 1090 PG II + 180 L UN 2904 | GPT-6 Astra / A | NO | YES | Model error. 240 × 3 + 180 × 1 = 900 points, which does not exceed 1000. Categories are from Table A and the multipliers from 1.1.3.6.4. Checked 2026-10-20. | model error |
adr_exemption-027145 L UN 1173 + 265 L UN 2219 | GPT-6 Astra / A | 700 | 1230 | Model error. 145 × 3 (category 2) + 265 × 1 (category 3) = 700. Checked 2026-10-20. | model error |
adr_table_a-011UN 3320 PG III, col 3b | GPT-6 Astra / A | C5 | CW1 | Model error. Table A gives C5. Checked 2026-10-20. | model error |
adr_table_a-026UN 2742, col 4 | GPT-6 Astra / A | II | I | Model error. Table A gives II. Checked 2026-10-20. | model error |
adr_table_a-035UN 3330, col 15 | Grok 4.7 / A | (E) | C | Model error. Re-read 2026-10-21: the workbook row gives "0 (E)". | model error |
adr_table_a-037UN 2336, col 15 | GPT-6 Astra / A | (C/E) | C/D | Model error. Re-read 2026-10-21: the workbook row gives "1 (C/E)". | model error |
adr_table_a-037UN 2336, col 15 | Grok 4.7 / A | (C/E) | D/E | Model error. Re-read 2026-10-21: the workbook row gives "1 (C/E)". | model error |
adr_table_a-039UN 1418 PG I, col 15 | GPT-6 Astra / A | (E) | B/E | Model error. Table A gives (E). Checked 2026-10-20. | model error |
adr_table_a-040UN 3374, col 15 | GPT-6 Astra / A | (D) | B/D | Model error. Re-read 2026-10-21: the workbook row gives "2 (D)". | model error |
adr_table_a-040UN 3374, col 15 | Grok 4.7 / A | (D) | B/D | Model error. Re-read 2026-10-21: the workbook row gives "2 (D)". | model error |
adr_table_a-044UN 1418 PG II, col 7a | GPT-6 Astra / A | 0 | 500 g | Model error. Table A gives 0 (LQ not permitted). Checked 2026-10-20. | model error |
identifiers-011AWB prefix 350 | GPT-6 Astra / A | YP | YN | Model error. The site's record is Air Premia (YP/350), sourced from Air Premia Cargo, whose site answered 403 on the day. ShipsGo also lists Air Premia as YP/350 (shipsgo.com), but it is an aggregator and counts only as support. YN is Air Creebec, which has no AWB prefix in the dataset. | model error |
identifiers-031USDVA | GPT-6 Astra / A | Delevan | Davant | Model error. USDVA is Delevan, NY in UN/LOCODE 2025-1 (site's parse). | model error |
identifiers-032JPIHO | GPT-6 Astra / A | Ihota | Iho | Model error. JPIHO is Ihota in UN/LOCODE 2025-1 (site's parse). Checked 2026-10-20. | model error |
identifiers-037JPYSS | Grok 4.7 / A | Yoshiase | Yatsushiro | Model error. JPYSS is Yoshiase, subdivision 34. Yatsushiro is a different code, JPYAT (subdivision 43). Both from UN/LOCODE 2025-1 (site's parse). | model error |
adr_2027_delta-005Under the ADR 2027 amendments, does Table A column (6) (special provisions) change for UN 3536 (LITHIUM BATTERIES INSTALLED IN CARGO TRANSPORT UNIT lithium ion | Claude Fable 5.1 / A | 389 410 | NO CHANGE | Model error: the site matches ECE/TRANS/WP.15/274 pp. 15–16 (the Table A instruction for this UN) Source: UNECE; ECE/TRANS/WP.15/274, /Add.1, /Corr.1 (accepted, C.N.395.2026; applies from 1 January 2027); record UN 3536 column (6); WP.15/274 p.16 | model error |
adr_2027_delta-009Under the ADR 2027 amendments, does Table A column (20) (hazard identification number) change for UN 1041 (ETHYLENE OXIDE AND CARBON DIOXIDE MIXTURE with more t | Claude Fable 5.1 / A | 238 | NO CHANGE | Model error: the site matches ECE/TRANS/WP.15/274 pp. 15–16 (the Table A instruction for this UN) Source: UNECE; ECE/TRANS/WP.15/274, /Add.1, /Corr.1 (accepted, C.N.395.2026; applies from 1 January 2027); record UN 1041 column (20); WP.15/274 p.15 | model error |
adr_2027_delta-011Under the ADR 2027 amendments, does Table A column (5) (danger labels) change for UN 1041 (ETHYLENE OXIDE AND CARBON DIOXIDE MIXTURE with more than 9 % but not | Claude Fable 5.1 / A | 2.1 +8 | NO CHANGE | Model error: the site matches ECE/TRANS/WP.15/274 pp. 15–16 (the Table A instruction for this UN) Source: UNECE; ECE/TRANS/WP.15/274, /Add.1, /Corr.1 (accepted, C.N.395.2026; applies from 1 January 2027); record UN 1041 column (5); WP.15/274 p.15 | model error |
adr_2027_delta-012Under the ADR 2027 amendments, does Table A column (3b) (classification code) change for UN 1041 (ETHYLENE OXIDE AND CARBON DIOXIDE MIXTURE with more than 9 % b | Claude Fable 5.1 / A | 2FC | NO CHANGE | Model error: the site matches ECE/TRANS/WP.15/274 pp. 15–16 (the Table A instruction for this UN) Source: UNECE; ECE/TRANS/WP.15/274, /Add.1, /Corr.1 (accepted, C.N.395.2026; applies from 1 January 2027); record UN 1041 column (3b); WP.15/274 p.15 | model error |
adr_2027_delta-017Under the ADR 2027 amendments, does Table A column (3b) (classification code) change for UN 3300 (ETHYLENE OXIDE AND CARBON DIOXIDE MIXTURE with more than 87 % | Claude Fable 5.1 / A | 2TFC | NO CHANGE | Model error: the site matches ECE/TRANS/WP.15/274 pp. 15–16 (the Table A instruction for this UN) Source: UNECE; ECE/TRANS/WP.15/274, /Add.1, /Corr.1 (accepted, C.N.395.2026; applies from 1 January 2027); record UN 3300 column (3b); WP.15/274 p.16 | model error |
adr_2027_delta-021Under the ADR 2027 amendments, does Table A column (5) (danger labels) change for UN 3300 (ETHYLENE OXIDE AND CARBON DIOXIDE MIXTURE with more than 87 % ethylen | Claude Fable 5.1 / A | 2.3 +2.1 +8 | NO CHANGE | Model error: the site matches ECE/TRANS/WP.15/274 pp. 15–16 (the Table A instruction for this UN) Source: UNECE; ECE/TRANS/WP.15/274, /Add.1, /Corr.1 (accepted, C.N.395.2026; applies from 1 January 2027); record UN 3300 column (5); WP.15/274 p.16 | model error |
adr_2027_delta-025Under the ADR 2027 amendments, does Table A column (20) (hazard identification number) change for UN 1727 (AMMONIUM HYDROGENDIFLUORIDE, SOLID)? Its ADR 2025 val | Claude Fable 5.1 / A | 86 | NO CHANGE | Model error: the site matches ECE/TRANS/WP.15/274 pp. 15–16 (the Table A instruction for this UN) Source: UNECE; ECE/TRANS/WP.15/274, /Add.1, /Corr.1 (accepted, C.N.395.2026; applies from 1 January 2027); record UN 1727 column (20); WP.15/274 p.15 | model error |
adr_2027_delta-028Under the ADR 2027 amendments, does Table A column (20) (hazard identification number) change for UN 2372 (1,2-DI-(DIMETHYLAMINO) ETHANE)? Its ADR 2025 value is | Claude Fable 5.1 / A | 368 | NO CHANGE | Model error: the site matches ECE/TRANS/WP.15/274 pp. 15–16 (the Table A instruction for this UN) Source: UNECE; ECE/TRANS/WP.15/274, /Add.1, /Corr.1 (accepted, C.N.395.2026; applies from 1 January 2027); record UN 2372 column (20); WP.15/274 p.15 | model error |
adr_2027_delta-030Under the ADR 2027 amendments, does Table A column (7a) (limited quantity) change for UN 2956 (5-tert-BUTYL-2,4,6-TRINITRO-m-XYLENE (MUSK XYLENE))? Its ADR 2025 | Claude Fable 5.1 / A | 0 | NO CHANGE | Model error: the site matches ECE/TRANS/WP.15/274 pp. 15–16 (the Table A instruction for this UN) Source: UNECE; ECE/TRANS/WP.15/274, /Add.1, /Corr.1 (accepted, C.N.395.2026; applies from 1 January 2027); record UN 2956 column (7a); WP.15/274 p.16 | model error |
adr_table_a-037what is the tunnel restriction code (Table A column 15) for UN 2336 (ALLYL FORMATE)? | Claude Fable 5.1 / A | (C/E) | D/E | Model error: the site matches ADR2025_tabA_E.xlsx (UNECE Digital ADR 2025, Table A) Source: UNECE; ADR 2025 (ECE/TRANS/352); record 2336_0; field tunnel_restriction_code | model error |
adr_table_a-039what is the tunnel restriction code (Table A column 15) for UN 1418, packing group I (MAGNESIUM POWDER or MAGNESIUM ALLOYS POWDER)? | Claude Fable 5.1 / A | (E) | B/E | Model error: the site matches ADR2025_tabA_E.xlsx (UNECE Digital ADR 2025, Table A) Source: UNECE; ADR 2025 (ECE/TRANS/352); record 1418_0; field tunnel_restriction_code | model error |
adr_table_a-039what is the tunnel restriction code (Table A column 15) for UN 1418, packing group I (MAGNESIUM POWDER or MAGNESIUM ALLOYS POWDER)? | Gemini 4 Argon / A | (E) | B/E | Model error: the site matches ADR2025_tabA_E.xlsx (UNECE Digital ADR 2025, Table A) Source: UNECE; ADR 2025 (ECE/TRANS/352); record 1418_0; field tunnel_restriction_code | model error |
adr_table_a-040what is the tunnel restriction code (Table A column 15) for UN 3374 (ACETYLENE, SOLVENT FREE)? | Claude Fable 5.1 / A | (D) | B/D | Model error: the site matches ADR2025_tabA_E.xlsx (UNECE Digital ADR 2025, Table A) Source: UNECE; ADR 2025 (ECE/TRANS/352); record 3374_0; field tunnel_restriction_code | model error |
adr_table_a-040what is the tunnel restriction code (Table A column 15) for UN 3374 (ACETYLENE, SOLVENT FREE)? | Gemini 4 Argon / A | (D) | B/D | Model error: the site matches ADR2025_tabA_E.xlsx (UNECE Digital ADR 2025, Table A) Source: UNECE; ADR 2025 (ECE/TRANS/352); record 3374_0; field tunnel_restriction_code | model error |
adr_table_a-044what is the limited quantity (Table A column 7a); answer 0 if limited quantity is not permitted for UN 1418, packing group II (MAGNESIUM POWDER or MAGNESIUM ALL | Claude Fable 5.1 / A | 0 | 500 g | Model error: the site matches ADR2025_tabA_E.xlsx (UNECE Digital ADR 2025, Table A) Source: UNECE; ADR 2025 (ECE/TRANS/352); record 1418_1; field limited_quantity | model error |
adr_table_a-053what is the danger label(s) (Table A column 5), primary first, joined with + for UN 3536 (LITHIUM BATTERIES INSTALLED IN CARGO TRANSPORT UNIT lithium ion batter | Gemini 4 Argon / B | 9 | 9A | Model error: the site matches ADR2025_tabA_E.xlsx (UNECE Digital ADR 2025, Table A) Source: UNECE; ADR 2025 (ECE/TRANS/352); record 3536_0; field labels | model error |
identifiers-005Which airline uses the air waybill (AWB) prefix 607? | Claude Fable 5.1 / A | EY | DN | Model error: 607 is Etihad (the record is sourced from Etihad Cargo) Source: Etihad Cargo; status_checked_at 2026-08-11; record etihad-airways; field awb_prefix | model error |
identifiers-031Which place does the UN/LOCODE USDVA designate? | Claude Fable 5.1 / A | Delevan | Danville | Model error: the site's UNECE 2025-1 record is Delevan, NY (42.48 N, 78.48 W) Source: UNECE; UN/LOCODE 2025-1; record USDVA; field name | model error |
identifiers-035Which place does the UN/LOCODE EEMRS designate? | Gemini 4 Argon / A | Meeruse | Mereküla | Model error: Meeruse (59.45 N, 24.67 E) in the site's UNECE 2025-1 record; Mereküla is a different place Source: UNECE; UN/LOCODE 2025-1; record EEMRS; field name | model error |
identifiers-040Which place does the UN/LOCODE JPKHR designate? | Gemini 4 Argon / A | Kihara | Kohari | Model error: the site's UNECE 2025-1 record is Kihara Source: UNECE; UN/LOCODE 2025-1; record JPKHR; field name | model error |
Rows answered in the earlier run and carried into this one were checked then, against the same sources; the table names the source for each. Questions are shortened; the full wording is in the repository.
ADR 2027: the gap in our own tools
30 questions ask what the ADR 2027 amendments change in Table A. Claude Fable 5.1 alone answered 16 of 30 correctly. With the FreightUtils tools it answered 1 of 30 and said UNKNOWN to 29: the tools serve ADR 2025. GPT-6 Astra, Gemini 4 Argon, Grok 4.7 said UNKNOWN to all 30, in every condition they ran.
That is a gap in FreightUtils: no FreightUtils tool serves ADR 2027 yet. The amended values are on the ADR 2027 changes pages, not in the API. An ADR 2027 field on adr_lookup is the next tool change.
ADR 2027 was accepted: depositary notification C.N.395.2026 (9 October 2026) records no objection to the amendments proposed by C.N.252.2026. It applies from 1 January 2027, and ADR 2025 may still be applied until 30 June 2027 (1.6.1.1). ADR 2025 remains the edition FreightUtils serves and the one that applies until ADR 2027 takes effect on 1 January 2027; after that, ADR 2025 may still be applied until 30 June 2027 (1.6.1.1).
| Model | Condition | Right | Unknown | Confident wrong |
|---|---|---|---|---|
| Claude Fable 5.1 | A, alone | 16 of 30 | 5 of 30 | 9 of 30 |
| Claude Fable 5.1 | B, with the tools | 1 of 30 | 29 of 30 | 0 of 30 |
| GPT-6 Astra | A, alone | 0 of 30 | 30 of 30 | 0 of 30 |
| GPT-6 Astra | B, with the tools | 0 of 30 | 30 of 30 | 0 of 30 |
| Gemini 4 Argon | A, alone | 0 of 30 | 30 of 30 | 0 of 30 |
| Gemini 4 Argon | B, with the tools | 0 of 30 | 30 of 30 | 0 of 30 |
| Grok 4.7 | A, alone | 0 of 30 | 30 of 30 | 0 of 30 |
Method
- Questions. 300 questions in six families, generated from the data FreightUtils serves and checked live against production before the set was frozen (20 October 2026; question set v0.2, sha256
43570a01fbf479f9aedcd391384276d773ab731de90cb208a73b7b7422bd0c5c). - Conditions. A: the model alone. B: the model with the FreightUtils MCP tools (24 of them, as the hosted server’s tool list returned them;
get_subscribe_linkleft out), up to 5 tool calls per question. One repeat. - Prompts. Fixed, one per condition:A: You are answering a freight reference question. Answer from your own knowledge; if you do not know, say UNKNOWN. Give the answer in the format the question asks for, with nothing else.B: You are answering a freight reference question. Use the FreightUtils tools provided to look the answer up or compute it before you answer. Answer with the value only, in the format the question asks for, with nothing else. If the tools do not give you the answer and you do not know it, say UNKNOWN.
- Models and settings. GPT-6 Astra: Temperature is refused by gpt-6-astra (HTTP 400 "Unsupported parameter: 'temperature' is not supported with this model"), so it runs at the provider default; reasoning effort left at the default. Claude Fable 5.1: Temperature is refused by claude-fable-5-1 (HTTP 400 "`temperature` is deprecated for this model"), so it runs at the provider default; adaptive thinking is always on and effort is left at the default (high). Gemini 4 Argon: temperature 0. Grok 4.7: temperature 0. Prompt caching was on wherever the provider offers it.
- Scoring. Deterministic, by the harness’s scorer against the expected value; no model judges another. Answers were collected from 20 October 2026 to 21 October 2026; rows already answered on 20 October 2026 were carried over and re-scored under v0.2.
- Cost. Prices read on 20 October 2026, from each provider’s pricing page; spend counted from each API’s usage fields. USD 23.89 for this run, USD 64.91 across all runs of the eval.
Limits
- One repeat per question: run-to-run variation is not measured.
- One tool set: these are FreightUtils tools, so condition B measures them and no other reference source.
- The eval’s author also serves the data. The expected values come from the same data the tools serve, which is why every disputed answer was checked against the primary source.
- HS structure is a floor, not a test of classification: it asks which chapter or heading a code sits in.
- A web-search comparison (the model with its provider’s web search and no FreightUtils tools) is not run yet.
Results commit
0fef2a8 (results/2026-10-21-full-pass) · question set v0.2 sha256 43570a01fbf479f9aedcd391384276d773ab731de90cb208a73b7b7422bd0c5c · read on 27 October 2026. The numbers on this page are rendered from the committed results; nothing is typed by hand. How FreightUtils sources its data is on the Methodology page.