Designing a coding agent pilot that compares tools on the same work

Why one team per tool tells you little

The obvious pilot gives Tool A to the payments team and Tool B to the platform team, then compares merged pull requests after a month. But the teams differ in codebase, test coverage, seniority, task mix and review strictness, and any of these can move the numbers more than the tool does.

GitHub makes the same point about its own Copilot impact dashboard: its adoption multiplier compares different groups of users, so team composition, project complexity and seniority can sway it as much as adoption. Assigning tools by team inherits that problem.

Two designs that keep the work constant

Random assignment of live tasks

Each team builds a pool of tasks it would do anyway, and each task is assigned to a tool at random before work starts, ideally within each task type. METR used this design in its 2025 study of experienced open-source developers, randomly assigning each real issue to allow or disallow AI tools, so both conditions drew on the same people and the same kind of work.

Replay of merged changes

Alongside the live tasks, replay recently merged changes. Each run works in an isolated copy of the repository whose history ends at the commit before the change, with no later commits, remote refs, caches or files from earlier attempts, and no network route to the merged pull request. The agent gets the task description and the existing test suite; the acceptance tests added with the change are held out and run afterwards. Replay compares correctness and cost on identical inputs, but it says nothing about review burden on new work.

Confounders to settle before day one

The sources of unfairness below are predictable. Write down how the pilot handles each one before results arrive.

• Task selection: in METR's 2026 follow-up, 30 to 50 percent of surveyed developers said they had held back tasks they did not want to do without AI. Fix the pool before assignment and log every withdrawn task with its reason.
• Unequal setup: if one tool gets careful repository instructions and a working test command and the other does not, you are comparing setups. Give every candidate the same instructions, commands and permissions, and record tool version, model and settings for every run.
• Self-reported speed: participants in METR's 2025 study expected AI to make them 24 percent faster and afterwards believed it had made them 20 percent faster, while the measured result was 19 percent slower. METR now calls those results out of date, but the gap between perception and measurement still applies to any pilot.
• Parallel sessions: engineers who run several agents at once make time per task unreliable, as METR also reports. In the live arm, use cost per accepted change and review minutes as the main efficiency measures.

A worked plan for comparing two tools

This plan is hypothetical: two product teams compare two coding agents over six weeks. The numbers are planning assumptions, not results.

1. 1

   Build the pools

   Each team lists 24 backlog tasks: eight bug fixes, eight test coverage tasks and eight small features, each with a written description and a way to verify it.

2. 2

   Assign at random

   Within each team and task type, a script assigns four tasks to Tool A and four to Tool B, and the assignment is committed before anyone starts.

3. 3

   Equalise the setup

   Both tools get the same build and test commands, review conventions, permissions and network access, each in its own instruction format.

4. 4

   Run the replay

   Thirty changes merged last quarter are replayed three times per tool, each run in a fresh sanitised copy of the repository, with the tests added by each change held out for grading. Report the pass rate on those held-out tests and the cost per passing replay task, failed attempts included, per task type.

5. 5

   Review as usual

   Agent pull requests go through normal review and required checks. Reviewers note review minutes and rework, and whether the change meets the live acceptance definition.

Write the decision rule before the first run

GitHub's guidance on piloting a Copilot feature or model recommends defining what a go looks like before the start and running for at least one full billing cycle, typically four to six weeks. A comparison between tools needs the same discipline.

• Two definitions of success: a replay task passes when the held-out tests and the existing suite pass and the diff stays in scope; a live change is accepted when it passes required checks and review and is merged without rework inside an agreed window.
• Separate measures: cost per passing replay task, and cost per accepted live change with review minutes and rework included. Never add the two together.
• A practical threshold: the smallest difference that would change your decision, for example ten percentage points of pass rate, a hypothetical figure. Compare tools case by case on the same replay tasks, run each several times, and if the variation across cases and trials is as large as the threshold, record the result as inconclusive.

Inconclusive is a legitimate outcome, because agent output varies between runs; Anthropic's evaluation guidance treats repeated trials as the normal way to measure an agent. A tie under your threshold means deciding on other grounds, such as cost or fit with your controls.

Where this design reaches its limits

Random assignment needs enough comparable tasks per type. If too few accumulate during the pilot, the live comparison is inconclusive, and the report should say so. Replay and live evidence answer different questions: replay covers correctness and cost on known changes, the live arm covers review burden and fit on new work. Reviewers may know which tool wrote a pull request, so keep the live acceptance definition mechanical.

When security requirements, data residency or your git host leave only one viable tool, skip the comparison and pilot that tool against the team's own baseline. A first 30 minute call with one of our engineers about either design needs no system access or production data.

Reading Copilot pilot metricsEvaluations on your repositories