Benchception
Every RL environment is itself a task, so which frontier model is best at building them? Benchception makes environment-authoring the thing under test: models compete not by answering, but by teaching.
For this demo, our golden environment is Supply Chain Bench (code), a HUD benchmark we hold out from every model. The contest runs on SupChain-Bench Verified: the 33 tool questions that survived verification against the real grader.
No contamination. The golden environment is never shown to either author or student during authoring or training; it is revealed only at evaluation. Models can't memorize the test; they can only learn to teach.
All three students tied
We ran the full pipeline end to end: both authors generated environments, both trained a Qwen-8B, and all three students played the held-out Supply Chain Bench. They finished in a dead heat.
When we looked closer, the environments the models authored were low-quality and, honestly, a little sloppy, so neither trained student learned much the untrained baseline didn't already know.
The tie is the signal. A flat leaderboard isn't a broken experiment; it is evidence that authoring a good RL environment is genuinely hard. The models fail for a valid reason, which is exactly what a strong benchmark should expose.
From blocks to a spec
Build is a Scratch-style canvas. We drag four kinds of blocks out of the tray (Environment, Tool, Task, Train), snap detail blocks into them, and describe each in plain language. The blocks form a recursive tree that the builder projects into a plain-language description, written to help a model rebuild Supply Chain Bench. That description, not code or JSON, is the only thing each model receives.
How we'd improve it
Because every environment is a task in itself, the breadth and integrity of the golden set is the whole experiment. We're really measuring how well models build HUD tasks.
Harden the golden set
Broaden and diversify the golden environments so no single benchmark can be gamed. Variety is the defense against contamination and overfitting.
Harnesses & visibility
Add inspection harnesses into the environments models generate, surfacing reward hacks and degenerate setups before any training run begins.
Paper → environment
An automated pipeline that turns any research paper into a golden environment on HUD, scaling the golden set far beyond what we can hand-author.
Human verification
Human-in-the-loop review on every generated golden environment, guaranteeing the dataset stays correct, fair, and trustworthy.