AI Agent Evaluation: The 56% Pass Ceiling

AI agent evaluation breaks down the moment tasks stop being one-shot, because a question-answering score says nothing about whether an agent can carry state through a forty-step business workflow. A new benchmark makes that gap measurable: on RealReplicaBench, the strongest model tested completes 60 of 107 real commerce and operations tasks, or 56.1%.

That number is the story. These are not trick problems. They are product publishing, freight booking, storefront configuration, spreadsheet and document production, supplier research, and logistics, run inside stateful replicas of the software people actually use. The best available agent finishes just over half of them, and the second-place model finishes fewer than 52%.

Key Takeaways:

• RealReplicaBench runs 107 tasks: 53 CLI, 28 browser, 16 file, and 10 API or MCP tasks, each in a fresh container with its own verifier.
• Top pass rate is 56.1% on the OpenClaw harness and 61.7% on the Accio harness, both by Claude Opus 5.
• The same model scores up to 7.4 points differently depending on which harness runs it, which means benchmark numbers partly measure the harness, not the model.
• Cost per task spans 81x across the board: DeepSeek V4 Flash reaches 76% of the top pass rate for about 1.2% of the price.
• The maintainers state plainly that raw task-level result bundles are not yet publicly checksummed, so the board is an audited aggregate rather than a standalone reproduction package.

What one-shot AI agent evaluation misses

Most agent benchmarks grade a single exchange. Ask a question, score the answer, move on. That format made sense when the thing being tested was knowledge.

It measures almost nothing about agency. Real work has state: you create a draft listing, then edit it, then publish it, and the third step only makes sense if the first two left the system in the right condition. It has recovery: something fails at step nine and the agent either notices or produces a confidently wrong final answer. It has duration: forty tool calls is a different problem from four, because errors compound.

RealReplicaBench is built by the Accio team at Alibaba International around that distinction. Its tasks require operating interfaces and changing state, and each one is graded by its own verifier, either deterministic or LLM-assisted. A task passes only when every required check passes, which is a stricter rule than partial credit.

What this proved: long-horizon and one-shot evaluation are measuring different capabilities, and only one of them predicts production behavior. What it left unsolved: stricter grading makes cross-benchmark comparison harder, because a 56% here is not a 56% anywhere else.

What RealReplicaBench actually runs

The suite is 107 tasks, and the composition tells you what it cares about.

The environments are local mock services modeling SaaS, commerce, messaging, document, and operational systems, so no production accounts are needed and the state resets cleanly. The public showcase renders 104 pages across eight mock services, including a product publishing flow, a freight booking search, and a storefront theme customization screen.

That design choice is the expensive part and the reason the benchmark is interesting. Testing an agent against the live internet gives you results you cannot reproduce next week. Testing it against a static transcript gives you results that do not involve agency. Replicas are the middle path, and they cost real engineering to build.

Every run preserves the resolved configuration, the trajectory, the verifier result, artifacts, logs, and container metadata, which is the difference between a benchmark you can audit and a screenshot of a number.

What this proved: reproducible state is achievable if you are willing to rebuild the software under test. What it left unsolved: mocks drift from the real services they imitate, and nothing in the design prevents that.

The scores: the best agent finishes just over half

Here is the top of the board on the OpenClaw harness, from the repository snapshot.

Twelve model families were run on OpenClaw and thirteen on the Accio harness, with published scores produced through Accio-managed evaluation endpoints using gemini-3.1-pro-preview as the judge.

The honest reading is not "agents are bad." It is that the frontier for autonomous multi-step business work sits near half. If you are planning to hand an agent a workflow unattended, the base rate says roughly one run in two needs a human. That is a useful planning number and it is very far from the demos.

It also compresses tightly. The gap between first and seventh place is 12 points, and models from four different labs sit inside a five-point band. Choosing a model is not where the leverage is.

The same model scores differently depending on the harness

This is the finding most worth carrying away, and it is visible only because the team published both harnesses side by side.

Same tasks, same verifiers, same model weights. Different scaffolding around them, and up to 7.4 points of difference.

That means an agent benchmark number is a property of a pair, not a model. The agent harness, meaning the loop that decides how tools are exposed, how errors are retried, how context is managed, and when to stop, contributes as much as a full model generation does. Claude Opus 4.7 on the better harness beats Claude Opus 4.8 on the worse one.

For anyone choosing infrastructure, this inverts the usual buying conversation. Teams spend weeks comparing models and minutes choosing the runtime around them, and the data here says the second decision is worth at least as much as the first.

What this proved: harness quality is a first-class variable in agent performance. What it left unsolved: there is no way yet to score a harness independently of the model running inside it.

Cost per task belongs in the result

The live leaderboard publishes an estimated cost per task alongside the pass rate, and that column reorders the entire buying decision.

DeepSeek V4 Flash reaches 76% of the top model's pass rate at roughly 1.2% of its cost. Put the other way: Claude Opus 5 costs about 81 times more per task to add 15 percentage points of completion.

Which of those is correct depends entirely on what a failure costs you. If a failed run means a human redoes twenty minutes of work, the cheap model wins by a wide margin even at a materially lower pass rate. If a failed run publishes a wrong price to a live storefront, it does not. A leaderboard sorted by accuracy alone hides that decision instead of informing it.

Two caveats on the cost column. It is described as an estimate, some entries are unavailable, and all models run at their API's default thinking effort. For non-multimodal models the image portion of vision tasks is handled by gemini-3-flash-preview acting as a vision assistant, so those runs are not a single model end to end.

What the benchmark admits it cannot yet prove

The reproducibility note in the README is unusually candid and deserves to be quoted rather than paraphrased: raw task-level result bundles are not stored in Git and do not yet have public immutable URLs or checksums, so the published board is an audited aggregate keyed by public result IDs, not a standalone reproduction package.

In plain terms, you can run the benchmark yourself with your own credentials, and you can read the published scores, but you cannot independently verify that the published scores came from the runs claimed. That is a normal state for a benchmark three days into public life, and it is a real limitation until it is fixed.

Two further caveats belong next to the numbers. The judge is a single model family, gemini-3.1-pro-preview, on LLM-assisted verifications, which introduces a known correlation risk. And the benchmark is published by a company that also sells an agent product, with its own harness scoring higher than the open source alternative in the same tables. Nothing in the repository suggests the tasks are tilted, and the tasks and verifiers are open for inspection, but the incentive is worth naming.

What this proved: the team is honest about the gap between an audited aggregate and a reproducible artifact. What it left unsolved: until result bundles are checksummed and public, the leaderboard is a claim rather than a proof.

How to read an agent benchmark before you buy anything

Three habits, all of which this board rewards.

Ask what the pass criterion is. Here it is every required check passing, which is why the numbers look low compared to benchmarks reporting partial credit. Average capacity, the macro mean of checks passed over checks total, sits around 0.86 for the top models, and the gap between 0.86 capacity and 56% pass is precisely the cost of "almost finished."

Ask which harness produced the number. After this dataset, a model score with no harness attached is incomplete information.

Ask what a run costs. Steps and tokens per task are the difference between a benchmark result and a budget.

If you are running agents on real workflows rather than evaluating them, the same logic applies to your own stack: the runtime around the model is doing more of the work than the model choice suggests. RealReplicaBench uses the open source OpenClaw harness as its reference, and MoClaw runs hosted OpenClaw agents, which is one way to stop maintaining that layer yourself. The benchmark scores models, not vendors, and no MoClaw configuration appears in these tables.

Half of real work is the number to plan around

The most useful takeaway from RealReplicaBench is not a ranking. It is a planning constant. On long-horizon, stateful, real-interface business tasks, the frontier today lands between 56% and 62% depending on the scaffolding, models from four labs cluster within a few points of each other, and the runtime you wrap around them moves the result as much as a model generation does.

Design for that. Agents that check in, agents whose failures are loud, and workflows where a human reviews the half that needs it will outperform any architecture built on the assumption that the pass rate is about to reach 95%. It is worth revisiting when the next frontier model lands, because this is a board that will move.

Verified against the GitHub API and repository on August 5, 2026: Accio-org/RealReplicaBench, Apache-2.0, created August 2, 2026, 431 stars, release v1.3.1. Scores are the repository snapshot; the live leaderboard is the source of record.

FAQ

How do you evaluate an AI agent?

By running it against tasks that require multiple steps and change state, then checking the resulting state with a verifier rather than grading the text it produced. One-shot question answering does not test agency.

What is a long-horizon agent task?

A task requiring many dependent steps, often dozens of tool calls, where later steps depend on the state left by earlier ones. Publishing a product listing, booking freight, or reconciling a spreadsheet against a system of record are typical examples.

What is RealReplicaBench?

A 107-task benchmark from the Accio team at Alibaba International that evaluates agents inside stateful, reproducible local replicas of real commerce and business software, with per-task verifiers and preserved run artifacts.

Which model scores highest on RealReplicaBench?

Claude Opus 5, at 60 of 107 tasks (56.1%) on the OpenClaw harness and 66 of 107 (61.7%) on the Accio harness, in the repository snapshot dated August 2026.

Why do the same models score differently across harnesses?

Because the harness controls tool exposure, retry behavior, context management, and stopping conditions. Those choices change how many multi-step tasks survive to completion, and the published gap reaches 7.4 points.