BENCHMARKS
Benchmark methodology
Hyperlane benchmarks the whole agent, not a model: a fixed harness version, a versioned task suite, and recorded model and cost per attempt. We report pass rate, cost per resolved task, wall-clock time, and tokens per resolved task, alongside the raw per-task results. Methodology is published before numbers so results can be checked, reproduced, and compared fairly.
What we measure, how a task is run, and why a number without a harness version is marketing.
Explore
What we measure
Four numbers per suite: pass rate, cost per resolved task, wall-clock time per task, and tokens per resolved task. Pass rate alone hides cost, and cost alone hides failure. We keep the per-task rows so a reader can recompute every aggregate instead of trusting a headline.
How a task runs
Every attempt runs the real agent loop in an isolated machine, with a recorded harness version, model, optimizer policy, and budget. Retries count as attempts. A task passes only when its checks pass; a run that stops early is a failure, not a partial success. Evaluator errors are reported as errors, not folded into the pass rate.
Why methodology comes first
A pass rate without a harness version, a task suite, and a model is not comparable to anything. Vendor numbers drift as models change, and fixture suites are not official leaderboards. Publishing the method first makes the eventual numbers auditable, and makes it obvious when two results are not measuring the same thing.
Limitations
Our internal suites are fixtures built to exercise the loop, not official benchmark submissions. Sandbox image, network policy, and model version all affect results. We label fixture results as fixtures, report the model and date with every run, and will not present an internal number as a leaderboard score.
Reproduce or request the data
The task suite, harness version, and raw JSON are available on request, and we will publish per-run results once the suite is versioned and stable. If you are evaluating agents, ask every vendor the same four questions: which harness version, which model, which tasks, and what did a resolved task cost.
Frequently asked
Do you publish official SWE-bench scores?
Not yet. Our current suites are internal fixtures that exercise the agent loop. We label them as fixtures and do not present them as official leaderboard results. When we submit to a public leaderboard, the submission will be linked here.
Which model do you benchmark?
The harness is model-agnostic, so the model is a variable rather than a constant. Every attempt records the model and routing policy that served it, and results are reported per model rather than pooled into one misleading average.
Can I reproduce a result?
Yes, on request. We share the task suite, the harness version, the model configuration, and the raw per-task JSON so you can rerun the same work and compare like with like.
Why is there no headline score on this page?
Because a score without a harness version, task suite, model, and date cannot be compared to anything. We would rather publish a method you can check than a number you have to trust.
Last verified 11 Nov 2026.
Sources: Hyperlane docs Hyperlane pricing
Keep exploring
Delegate the ticket. Review the PR.
Start with the Free plan — 10 minutes of agent computer time every month, no card required.
Try Hyperlane freeHyperlane AI is not affiliated with the Hyperlane blockchain protocol (hyperlane.xyz) or the Hyperlane code editor (hyperlaneide.com).