SupChain-Bench Verified
A held-out benchmark is only as trustworthy as its ground truth. Before SupChain-Bench could be the golden environment in Benchception, it had to be gold-correct, ungameable, and gradable with no human in the loop. We audited all 326 items through the real graders and kept only the ones that survive.
SupChain-Bench (Guan, Liu & Cao, ACL 2026 Findings) models a three-tier supply chain with two tracks: QA answered from reasoning alone, and tool orchestration scored by Information Retrieval Accuracy (IRA) against a deterministic oracle. It is a strong benchmark, but it shipped as research code with no guarantee that every item was gold-correct or fairly gradable. For a held-out judge, that gap is disqualifying: any item where the gold is wrong, the question contradicts the data, or an efficient agent is penalized for unrequested work adds noise that looks exactly like model performance.
The exclusions are the product. Every excluded item is a landmine we took off the board.
How we verified
Two graders, nothing given the benefit of the doubt.
Put every QA gold to a blind panel
We kept a QA question only if its gold scored 1.0 through the real grader and survived a blind panel that never saw it: a Sonnet 3-vote pass plus six vendors answering cold (opus-4-8, sonnet-4-6, haiku-4-5, qwen3-max, deepseek-v3.1-terminus, gpt-5.1). Where 0 of 6 landed on the gold, we dropped it; where only some disagreed, we reviewed it manually.
9 excluded, 217 verified. Seven near-certain gold errors plus two that live agent runs scored 0.
Check every tool question for fairness
How the grader scopes ground truth decides fairness. Named-entity questions scope IRA to one entity, so they are fair on every track; whole-order questions are graded on the entire order and only the SOP prompt tells the agent to enumerate warehouses, so they are fair on the SOP track alone. We kept a question only if the oracle resolves the order, every named entity exists, there is a non-null field to grade, and the premise matches the data.
Asserts what the data denies (“why does this trade order have no fulfillments?” when it has five).
No resolvable order, or a named entity absent from the data.
Answerable at the buyer or fulfillment level, yet the whole-order grader still counts every warehouse field.
67 of 100 excluded, 33 verified. 19 named-entity questions (all tracks) plus 14 whole-order questions (SOP only).
Package the survivors on HUD v6
We rebuilt the survivors as two deployable environments, with 288 task instances routed automatically.
Live tools
The 8 tools run as an in-process MCP capability; the grader reconstructs the trajectory to compute IRA. No offline predictions, no manual scoring.
Deterministic grading
Reward is IRA (tool) or exact answer-set match (QA), with no LLM on the critical path. An optional LLM-judge is off by default and never touches reward.
SOP adherence is not graded. Reward depends only on the fields retrieved. Following the SOP is simply how you cover them, which is coverage, not obedience.
The decisions behind the number
Track fairness
Named questions run on all three tracks; whole-order questions run on SOP only, since anywhere else they would be unfair.
A redundant track, removed
The paper's ReAct-vote was byte-for-byte identical to ReAct on the platform, so we dropped it rather than inflate the count.
Every item gold-correct, fairly gradable, reproducible
288 verified instances from a 326-item corpus, every one gold-correct and fairly gradable, with a per-item exclusion reason recorded. The taskset and inventory regenerate byte-for-byte and exclusion decisions live in code, so the set is fully reproducible.
Where it goes next
Broaden the verified set
More orders, failure modes, and question shapes, since variety is the defense against gaming.
Automated re-verification
Re-run the full audit on every data or grader change, so the set cannot silently drift.
Reward-hack harnesses
Surface degenerate paths that score without doing the work, before any training run depends on them.
Human-in-the-loop review
A human verdict on the hardest exclusion calls as the set grows.
Built on the original SupChain-Bench (Shengyue Guan, Yihao Liu, Lang Cao), Apache-2.0. Our contribution is the verification layer and the HUD v6 packaging.