# SWE-bench Verified selection

`data/swe-bench.json` is generated by `scripts/import_swe_bench.py` from the official SWE-bench website repository's `data/leaderboards.json`. It includes only Verified runs using mini-SWE-agent, omitting warning-marked and unscored runs. All organizations in that source selection are retained. Rows count evaluations, not unique models. No name-based join to Epoch entries is performed; ECI, its comparison readout and scientific-discovery counts are unchanged.

## Reuse and attribution

Source: SWE-bench Team, [official leaderboard](https://www.swebench.com/) and [source data](https://github.com/SWE-bench/swe-bench.github.io/blob/master/data/leaderboards.json).

The website repository's root [LICENSE](https://github.com/SWE-bench/swe-bench.github.io/blob/master/LICENSE) is **CC BY-NC 4.0**, separately preserved in [swe-bench-license.txt](swe-bench-license.txt). This selection is for noncommercial reuse only; the site's catalog CC BY 4.0 and code MIT licenses do not replace these upstream terms. Retain source credit, license notices, links and transformation notices with redistributed copies. Obtain separate permission from the source for commercial use or a broader license.

Changes: select a benchmark and agent, omit flagged/missing-score runs, rename fields and sort rows. Scores are not recalculated. The JSON carries its attribution, license and changes. Recommended credit: “SWE-bench Team, SWE-bench website leaderboard, CC BY-NC 4.0; selection and field adaptation by AI Science Tracker.”

This component is not included in the current Hugging Face catalog export. Repository archives containing it, including Zenodo source archives, must retain its separate license and may not describe the entire archive as uniformly CC BY 4.0. Any future dataset distribution must identify this component separately; do not merge it into unrestricted catalog exports.

## Interpretation

The source evaluates 500 human-filtered issues. `score` is the reported percentage resolved; zero is valid and missing values are not zero. Preserve exact run IDs, agent versions, reasoning settings, source team check flags and evaluation dates. A source team check is not independent verification by this site. Versions 1.x and 2.x are not necessarily comparable, and settings/budgets may differ within versions. `mean_cost_usd` is the reported historical mean task cost, not a current price quotation. The newest agent version is selected initially; older configurations remain available. No inference is made about missing or newer models.

## Refresh

```sh
python3 scripts/import_swe_bench.py /private/path/leaderboards.json --fetched-on YYYY-MM-DD
```

Use the actual retrieval date. The daily refresh workflow downloads the official source, imports this separate JSON and runs the project checks. An unchanged normalized selection retains its earlier snapshot date. `selection_sha256` fingerprints the selected normalized records, not the full upstream file. Keep full downloads private. The initial snapshot was selected from official GitHub source blob `1609701d2d73b0f785d28d247155886062d942f1`.
