Methodology

What counts as an agent

An agent here is Model + Harness/scaffold: a system that takes multiple steps, uses tools, or interacts with an environment. A bare model answering one prompt is not in scope. This matters because the same model scores very differently under different harnesses, so every row records the system that holds the score, not just the model name.

A benchmark is included when it evaluates that kind of system. Meta-leaderboards that aggregate other benchmarks are included but marked tracker, because they are not a target to beat.

Saturation

sotaNorm expresses the state of the art as a percentage of the reference for that metric:

• Where the metric has a fixed ceiling — a percentage, a success rate — the reference is that ceiling, normally 100.
• Where it does not — dollars earned, hours of autonomy, steps completed — the reference is the measured human baseline. Without a human study, no saturation is computed and the row is marked nosota.

headroomNorm is simply 100 − sotaNorm: how many points of the metric are still unclaimed. Rows fall into four bands:

Saturation bands, as defined in schema.json

Band	Meaning
open	Below 50% of the reference — far from solved
active	50–75%
near	75–90% — near-saturated
saturated	90% or above
nosota	No single comparable state of the art
tracker	Meta-leaderboard, not a target

Gap to human

humanNorm normalises the human baseline the same way, and gap is sotaNorm − humanNorm in points. A positive gap means the agent is above the human reference on that metric.

Priority to beat

A composite score meant to answer “which benchmark is worth attacking next?”. Four components, each scored 1 to 5:

priority = 0.40 × p1  Headroom      how much room is left
         + 0.25 × p2  Durability    how slowly it is likely to saturate
         + 0.20 × p3  Credibility   adoption and trust in the result
         + 0.15 × p4  Feasibility   how cheaply it can be attempted

Tiers: High ≥ 3.6, Med 3.0–3.59, Low below 3.0. Trackers, pure-measurement and safety rows get no priority score, because “beating” them is not meaningful.

Where the figures come from

Every row carries a source URL pointing at the primary publication for that number — the benchmark's own leaderboard, the paper, or the vendor's announcement — and a notes field holding the caveats. Figures are transcribed, not reproduced: nothing here is re-run.

The date field records when the state of the art was set, which is frequently older than the snapshot date. as_of in data.json records the snapshot period the whole dataset describes.

What this dataset does not cover

The honest list. A report built on these numbers inherits all of it.

• No independent verification. Figures are transcribed from their sources. Several 2026-06 entries are vendor-reported, on harnesses that are not standardised across rows, and are marked as such in notes.
• Non-GA and withdrawn systems appear. Some rows are held by models that are restricted, preview-only, or were suspended after publication. Where the best generally-available figure differs, notes records both.
• No repeatability or variance data. One reported figure per row. No error bars, no re-runs, no seed sensitivity. Two benchmarks that differ by a point are not meaningfully distinguishable here.
• No cost normalisation. Cost appears only where a benchmark's own metric is denominated in dollars. There is no per-row cost-to-reach-SOTA, so rows cannot be compared on efficiency.
• No time-to-saturation history. Each row is a single point in time. Rate of saturation is not recorded, so trend claims cannot be drawn from one snapshot.
• Contamination is noted, not measured. Where a benchmark is known or suspected to be contaminated, notes says so. No contamination analysis is performed here.
• Coverage is curated and partial. Selection favours agentic benchmarks that are publicly reported and English-language. Absence from this list is not evidence a benchmark is unimportant.
• Normalisation loses information. Reducing dollars, hours and steps to a percentage of a human reference makes rows comparable at the cost of precision. Always read the raw sota, unit and ceiling alongside sotaNorm.

Corrections and versioning

The dataset carries an explicit version of the form YYYY.MM.PATCH. Figures behind a URL change; a version does not. Cite the version.

When a published figure turns out to be wrong it is logged on the corrections page with the reason and the date, and the patch version increments. Errors are recorded rather than quietly overwritten — a dataset that admits its mistakes is easier to trust than one that has apparently never made any.

How to cite

Machine-readable citation metadata lives in CITATION.cff and in the cite_as field of data.json. The current form is:

Santa María Morales, Santiago (2026). AI Agent Benchmarks —
Saturation Tracker (version 2026.08.0) [Data set].
https://benchmarks.santismm.com/

The dataset is licensed CC BY 4.0: reuse it freely, including commercially, as long as you credit it and link back.

Embedding the table

If you would rather show the data than copy it, embed.html renders a compact saturation table that reads the live dataset, so an embed can never display a figure that has already been corrected here. It carries its own attribution and licence link.

<iframe src="https://benchmarks.santismm.com/embed.html?n=10"
        title="AI agent benchmark saturation"
        width="100%" height="460" loading="lazy"
        style="border:1px solid #2a323d;border-radius:8px"></iframe>

Three parameters, all optional:

• n — how many rows, 1 to 50. Default 10.
• view — saturated (default) for the most saturated benchmarks, open for the ones with the most headroom left.
• group — restrict to one coarse domain, for example group=Software%20engineering. The available values are the group field in data.json.