Harvey LAB: An Open Legal Agent Benchmark

Harvey AI has open-sourced its legal agent benchmark, and the design choice that makes it interesting is the grading rule: a task passes only if every rubric criterion passes. Find eight of ten risks in a deal report and you score zero, not eighty percent. Their announcement puts it bluntly, saying a report that catches eight of ten risks is not 80% useful, it is materially incomplete.

LAB stands for Legal Agent Benchmark. It lives at harveyai/harvey-labs, MIT licensed, and the repo carried 870 stars as of August 10, 2026 after going up on March 30. Harvey is the legal AI company valued in the billions, so this is a market leader publishing the measuring stick rather than a research group proposing one.

Key Takeaways:

• Two parts ship together: a dataset of tasks (instructions, documents, rubrics) and an execution harness for running and scoring agents against them.
• The repo badge reads 1,671 tasks across 24 practice areas plus contracting. The announcement post says "more than 1,200," so the set has grown since launch.
• Rubric criteria are atomic and binary, covering facts, conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts and formatting. Harvey counts over 75,000 of them, expert-written.
• There is no leaderboard. Harvey says it launched without one deliberately and plans to publish results later. Any ranking you see quoted today isn't theirs.
• Tasks come from real client matters worked by practising lawyers, broken down into associate-level pieces.

What Harvey LAB actually contains

A benchmark is only as good as what it asks, and the shape of these tasks is the story.

The dataset side holds tasks made of three things: instructions telling the agent what the assignment is, the documents it has to work from, and the rubric it gets graded against. The harness side runs an agent through a task, collects the work product, applies the rubric, and produces reports and comparison dashboards. Model adapters let you plug in whatever agent you're testing.

The tutorial walks one realistic M&A data-room assignment from setup through task inspection, an agent run, scoring, report review and comparison. Anyone who has spent a week inside a data room will recognise why that example got chosen; it's the archetype of legal work that is enormous, tedious, structured, and unforgiving about the one clause you skipped.

Two numbers deserve a careful read. The badges in the README say 1,671 tasks and 24 practice areas plus contracting. The launch post says more than 1,200 tasks across 24 practice areas, graded by over 75,000 expert-written rubric criteria. Both are Harvey's own figures; the repo has simply moved since publication, which fits their statement that LAB is ongoing and the task set keeps growing. Cite the repo if you want the current count, the post if you want the launch baseline.

What it actually tests

Long-horizon work, which is a specific thing and not a synonym for hard.

The framing throughout is agents doing work that mirrors law-firm practice: open-ended assignments, context assembled across many matter files, and output that a partner would review. Not question answering. Not retrieval from a single contract. The unit of evaluation is a piece of work product an associate would hand in.

Because tasks derive from real client matters that practising lawyers handled, the messiness comes along for free. Real matters have documents that contradict each other, deadlines buried in exhibits, and dollar amounts that appear in three places with two of them stale. That's exactly where an agent that reads well but reasons shallowly falls apart, and it's the same failure mode the wider field has been circling in long-horizon agent evaluation.

The rubric criteria list is worth reading as a description of what legal work is: facts, conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts, formatting choices. Formatting sits in the same list as conclusions on purpose. A memo with the right analysis in the wrong shape still gets sent back.

The all-pass rule, and why it's the interesting part

Most benchmarks report partial credit, because partial credit produces a smooth number that moves as models improve, and a smooth number is what a leaderboard needs.

LAB refuses that. A task passes when every criterion passes. The effect is brutal on scores and honest about the job: nobody bills a client for a risk assessment that found most of the risks, and the missing one is the whole reason you hired a lawyer. Grading this way turns the benchmark into a proxy for "could this go out without rework," which is the only question a firm actually cares about.

It also makes the numbers unflattering, and Harvey seems to have accepted that. When scores do get published, expect them low. A benchmark where frontier models score in the teens is more useful right now than one where everything clusters at 85 and stops discriminating.

The scoring runs on rubrics with an LLM judge, documented in the repo's evaluation methodology guide. Judge-based scoring carries its own known problems, and the all-pass rule sharpens them: one judge misreading one criterion flips a whole task from pass to fail. Whether that noise washes out across 1,671 tasks is an empirical question nobody has published an answer to yet.

Vertical benchmarks are where evaluation is going

Two years ago the interesting agent benchmarks were general: can it browse, can it use a computer, can it finish a multi-step task in a sandbox. The results told you about capability in the abstract.

They stopped being decisive around the point where buyers started asking a different question. Not "is this agent capable" but "is this agent good enough at my work to replace a step in my process." No general benchmark answers that, and the gap between general competence and domain-acceptable output turns out to be enormous in exactly the fields where the money is: law, medicine, accounting, compliance.

Harvey building LAB from real matters is what that shift looks like in practice. A legal agent benchmark sits alongside general capability tests like ARC-AGI-style evaluations rather than competing with them; one tells you whether the model can reason, the other whether the agent can do the job. A vendor publishing the benchmark for its own market is not a neutral act, and the obvious question, whether a benchmark authored by Harvey happens to reward the way Harvey's product works, is fair to ask. Open-sourcing the tasks and rubrics is the only real answer to it, and they did that.

Running it yourself, and what that costs

Nothing stops a firm from running this internally, and that may end up mattering more than the eventual leaderboard.

The harness ships with model adapters, so you point it at whatever agent you're evaluating, run a sweep, and get reports plus comparison dashboards out the other end. A procurement team assessing three vendors has, for the first time, a legal agent benchmark it can run against all three under identical rubrics instead of comparing three vendor decks.

Costs land in two places. Every task means an agent run over a document set, which is not cheap on a long-horizon assignment, and every scored task means a judge pass over the rubric criteria. With over 75,000 criteria across the full set, running everything is a real bill. Start with one practice area.

There is no formal release on the repo, incidentally, only tags; the citation block points at v1.0. Pin a tag when you run it, because comparing a January score against an August score means nothing if the task set grew in between, and Harvey has said plainly that it will keep growing.

What this doesn't tell you

A legal agent benchmark measures work product against a rubric. It does not measure the things that decide whether a firm actually adopts anything.

It says nothing about whether the agent's output can be traced back to a source document when a partner asks. Nothing about privilege, confidentiality, or where the documents went during processing. Nothing about cost per matter, latency, or what happens when the agent is wrong in a way the rubric didn't anticipate. Those are the questions that stall pilots, and no benchmark score answers them.

It also can't tell you about the human side, which is where legal AI deployments usually break: who reviews the output, how long that review takes, and whether the reviewing associate ends up spending as long checking the agent as they would have spent doing the work. Measuring the agent alone measures half the system.

Read a score, when scores exist, as a floor rather than a verdict. An agent that fails LAB will fail in your matters too. An agent that passes has only proven it can produce work product that survives a rubric, which is necessary and nowhere near sufficient.

FAQ

Is Harvey LAB free and open source?

Yes, MIT licensed, with the task dataset and the execution harness both in the public repository. Running it costs you model inference for the agent under test plus whatever the LLM judge consumes during scoring.

What legal tasks does it evaluate?

Associate-level work product across 24 practice areas plus contracting, derived from real client matters. The published tutorial takes an M&A data-room assignment end to end. Tasks are long-horizon by design: open-ended instructions, context spread across many documents, and output meant to be reviewed rather than read as a chat reply.

Is Harvey LAB affiliated with Harvey AI?

It's Harvey's own project, published under the harveyai GitHub organisation with an announcement on harvey.ai. Not a third-party effort using the name.

Does Harvey publish a leaderboard of model scores?

Not at launch, and that was deliberate. Harvey said it was launching without a leaderboard and intends to publish one after input from research partners. As of August 2026 no official LAB scores exist, so treat any ranking attributed to this benchmark with suspicion.

How is scoring done?

Expert-written rubrics with atomic pass/fail criteria, applied by an LLM judge, under all-pass grading: a task counts as passed only when every criterion passes. Partial credit does not exist here, which is the point.