# Independent numerical benchmark evidence

These ten components are separate from the scientific-discovery catalog, ECI, the Epoch notable-model inventory and the existing SWE-bench Verified component. Numbers are reported by the sources, not independently reproduced here. Counts below are **observations or evaluation configurations**, not unique models. Initial integration: **18 November 2026**. The table below records the initial counts; current counts, accepted revisions and per-component snapshot dates are in the component registry and downloads.

## Source and reuse directory

| Component | Selected numerical evidence | Observations | License and attribution |
| --- | --- | ---: | --- |
| GPQA Diamond | All scored Epoch **own runs**, mean accuracy in the original 0–1 scale | 340 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); credit Epoch AI, Capabilities & benchmarking. [Source](https://epoch.ai/benchmarks/gpqa-diamond), [download](../data/benchmarks/gpqa.json), [license text](benchmark-licenses/cc-by-4.0.txt). |
| Arena | Text/latest, overall category; rating points, bounds and vote counts | 421 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); credit Arena / LMArena. [Licensed numerical dataset](https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset), [download](../data/benchmarks/arena.json), [license text](benchmark-licenses/cc-by-4.0.txt). |
| τ²-bench | Active text submissions, pass^1 per available domain; higher pass^k values retained in details | 54 | MIT; credit Sierra Research and result submitters. [Source](https://github.com/sierra-research/tau2-bench/tree/main/web/leaderboard/public/submissions), [download](../data/benchmarks/tau.json), [original copyright and license](benchmark-licenses/tau2-bench-license.txt). |
| ChemBench | Only aggregate fraction_correct from two historical repository reports, gpt-4o and claude3.5 | 2 | MIT; credit LAMA Lab / ChemBench authors. [Legacy reports](https://github.com/lamalab-org/chembench/tree/dev/reports), [download](../data/benchmarks/chembench.json), [original copyright and license](benchmark-licenses/chembench-license.txt). |
| BixBench v1.5 | All six released **zero-shot baseline** configurations, not paper agent runs | 6 | Apache-2.0; credit FutureHouse / BixBench authors. [Source](https://github.com/Future-House/BixBench/blob/main/bixbench-v1.5_results/zero_shot_baselines.json), [download](../data/benchmarks/bixbench.json), [license](benchmark-licenses/BixBench-license.txt). |
| MMLU-Pro | Historical README mini-leaderboard only | 17 | Apache-2.0; credit TIGER-AI-Lab, Yubo Wang and MMLU-Pro coauthors. [Source](https://github.com/TIGER-AI-Lab/MMLU-Pro), [download](../data/benchmarks/mmlu-pro.json), [license](benchmark-licenses/MMLU-Pro-license.txt). |
| CharXiv | All released stats aggregate evaluations; descriptive and reasoning validation separate; original reference labels retained | 78 | **CC BY-SA 4.0**, including this adapted numerical component; credit Princeton NLP / CharXiv authors. [Source](https://huggingface.co/datasets/princeton-nlp/CharXiv), [download](../data/benchmarks/charxiv.json), [original data license](benchmark-licenses/charxiv-data-license.txt), [full legal code](benchmark-licenses/cc-by-sa-4.0.txt). |
| RULER | All available model scores at six measured lengths, 4K–128K; absent values omitted | 264 | Apache-2.0; credit NVIDIA / RULER authors and contributing model authors identified in source notes. [Source](https://github.com/NVIDIA/RULER), [download](../data/benchmarks/ruler.json), [license](benchmark-licenses/RULER-license.txt). |
| AgentDojo | All numeric utility, utility under attack and targeted attack-success cells in repository results table | 84 | MIT; credit ETH Zurich SPY Lab / AgentDojo authors. [Source](https://github.com/ethz-spylab/agentdojo/blob/main/docs/results-table.html), [download](../data/benchmarks/agentdojo.json), [original copyright and license](benchmark-licenses/agentdojo-license.txt). |
| SWE-bench Pro | Seven **initial** SWE-Agent runs from committed HTML; 730 problems, uncapped cost, 250 turns | 7 | MIT; credit Scale AI / SWE-bench Pro authors. [Source](https://github.com/scaleapi/SWE-bench_Pro-os/blob/main/index.html), [download](../data/benchmarks/swe-bench-pro.json), [original copyright and license](benchmark-licenses/SWE-bench_Pro-os-license.txt). |

The [component registry](../data/benchmarks/index.json) provides exact revisions, selections, attribution, limits and output SHA-256 hashes. Each download repeats its component license, attribution, modifications and source hashes. There is no combined score or combined numerical dataset licensed under the catalog's blanket CC BY 4.0 grant. The current catalog Hugging Face builder exports discoveries and vulnerabilities only; these benchmark components are not added to that export. Repository releases may archive these files **with their component terms intact**.

## What changed and what must travel with reuse

We selected numerical results, renamed fields and labeled evaluation slices. Scores retain their original numeric scales and precision in JSON; the table rounds display values to four decimal places. Original inference labels and available source methodology are retained. We did not add model-name joins, recalculate accuracy, estimate new confidence bounds, or infer missing scores or evaluation dates. The source order is retained.

When redistributing a component, retain its attribution, source/revision links, license, disclaimers and change notices. CC BY components require appropriate credit, a license link and disclosure of modifications. The **CharXiv adapted numerical file remains CC BY-SA 4.0**: retain the same obligations and license adaptations under CC BY-SA 4.0, including applicable database adaptations. This obligation applies to that component; it does not relicense unrelated code or datasets merely collected beside it. Do not impose extra downstream restrictions. MIT components retain their original copyright and full permission/disclaimer notice. Apache components retain their license and applicable notices; our extraction/field changes are explicitly stated in every file. No source root NOTICE file was present in the reviewed repository listings. Preserve any additional notices introduced by future source revisions.

Questions, expected answers, paper charts, external copyrighted corpora, prompts, attack payloads, conversations, private data, contact details and trajectories are **not redistributed** in these components. The source licenses do not imply source-author endorsement. Linked model licenses describe those models, not the numerical dataset license.

## Interpretation limits

- GPQA configurations can use different reasoning settings and budgets. Source stderr is not a confidence interval. We import Epoch's own-run CSV only: its external SciCode/LiveBench/etc. imports retain external rights and are excluded.
- Arena measures preferences. Rating points are not percentages, and different categories or publication dates are not directly comparable. Source-provided rating bounds and vote counts remain in details.
- τ² text results are separated by domain, version, retrieval and simulator where reported. A slice is a viewing aid, not proof of identical settings. Other settings and source methodology notes may differ or apply only to one domain; consult the exact submission. Submission, evaluation and model-release dates are not interchangeable.
- ChemBench is a two-report **legacy selection**, not the current separately hosted ChemBench-Results feed. The source labels do not identify exact releases. BixBench zero-shot baselines are not agentic bioinformatics results.
- MMLU-Pro, CharXiv, RULER and AgentDojo contain historical models and protocols. CharXiv's archive was last modified in September 2024; we do not import newer website results. Reference controls such as Human or Random are not independent model releases.
- RULER claimed length is not effective length. Its original source notes attribute Jamba results to the contributing report, FILM to its paper (YaRN beyond 32K and no one-shot CWE example), and Qwen/EXAONE results to their respective papers. Original notes remain linked per row. This is the selected original README table, not RULER v2.
- AgentDojo attack success is **lower is better**; utility is **higher is better**. Attack and defense settings differ.
- SWE-bench Pro's selected source calls its scores **initial runs subject to change**. This collection is distinct from SWE-bench Verified (whose existing component is CC BY-NC 4.0) and from separate Scale commercial/private result feeds.

## Reproduction and maintenance

The existing **Daily refresh** workflow checks all ten selections every day at **04:30 UTC** (05:30 Zurich in winter, 06:30 in summer), alongside Epoch and SWE-bench Verified. `scripts/refresh_public_benchmarks.py` accepts source updates only within the stated selections and supported formats. It checks numerical-dataset license metadata, exact approved legal texts, new repository NOTICE files, complete pagination, stable HF revisions, numerical validity and selected protocol assumptions. It preserves attribution, source pins, hashes and component-specific dates. Unchanged numerical content does not advance a snapshot date or create a benchmark change. If a source is inaccessible or a rights/format/protocol check fails, its last accepted component is retained; other valid updates may proceed. A visible Actions warning, job summary and 30-day per-source report artifact record held sources. A changed license or unsupported format needs explicit review; the job does not guess permission or silently relicense data. Historical/preliminary scope still applies: checking an old archive daily does not make its evaluations recent. New external leaderboards and broader selections are not automatically admitted.

`scripts/public-benchmark-inputs.json` pins the reviewed raw inputs, Git commits/HF revisions, SHA-256 hashes and notices. Raw files stay in private `work/`; they must not be committed. Rebuild against the retained input cache:

```sh
python3 scripts/import_public_benchmarks.py --inputs work/metric-enrichment
```

For available remote inputs, `--download` retrieves missing pinned files and refuses hash mismatches. Arena uses the complete 11,177-row frozen text/latest Parquet file before selecting the 421 overall observations; the HF revision is checked unchanged across retrieval. This avoids repeatedly requesting the paginated dataset-viewer API. Install the build-time reader with `python3 -m pip install -r scripts/requirements-benchmarks.txt`; Apache Arrow is a build dependency only, not website JavaScript. **The Epoch ZIP URL is mutable**: if it changes, the old source cache or the appropriate archived upstream export is needed to reproduce the old snapshot. The importer stops rather than silently replace it. A new date/revision is a new reviewed snapshot, not a refresh of the old date.

The importer writes `data/benchmarks/*.json`; do not edit those outputs manually. License notices in `docs/benchmark-licenses/` are exact copies of the reviewed upstream texts. The browser loads one collection at a time from local JSON, uses no external libraries or runtime requests to third parties, and offers source-specific downloads even when a numerical fetch fails.
