ServicesCoding agents
Coding agent evaluations on work you have already merged
Before you pick a tool, a model or a setting, we replay changes from your own repositories against tests the agent can't see, and compare cost, acceptance and review effort. Then we check the result on live work.
What happens in a replay
- 1
Choose past changes
Tasks from your history that reflect your usual work
- 2
Reset the repository
Each candidate begins at the commit just before that change
- 3
Let each candidate work
An identical task for each, logged with its tool, model and settings
- 4
Keep the tests hidden
The tests that came with the change are withheld from the agent
- 5
Compare the outcomes
Which changes are accepted, what reviewers flag and what it costs to get one accepted
What each run is measured on
Three things define a run: the tool, the model behind it and how they are configured. We log all three. Where the tool permits, we change only one of them at a time.
Alongside these, we count retries, reverts, rework and review findings. We report results per task type. That shows when one candidate is best at bug fixes and another at refactoring.
How we define an accepted change
Acceptance is agreed with you before anything runs. A change counts if it passes the held-out tests, keeps to the task's scope and could be merged by a reviewer as it is. Review effort is measured as the time a reviewer spends on each change. The cost of an accepted change includes every retry and every abandoned run. So a candidate that is cheap per run but needs three attempts is charged for all three.
Confirming results on live work
A replay alone never decides. One group of tasks stays out of all tuning, and we confirm each result on tasks your engineers are working on now before anyone relies on it. That covers the two ways a replay can mislead: future work that differs from past tasks, and changes that pass every test and are still wrong.
Deciding on new models and harness changes
When a new model comes out or your harness changes, the same setup gives you evidence on whether to roll the change out further. Because we test each change on its own and keep a record of every run, your team can rerun the evaluation by itself.
Testing changes to a workflow
The method also applies when what changes is the way a tool is used, not the tool itself. You can run a modified pstack skill or playbook next to your base setup, with one set of scoped tasks for both. The same goes for two Kilo configurations that differ in provider or model. Devin Desktop sessions can be compared with work handed off to the cloud only if the product can begin on a restored checkout. The tests must stay outside that checkout. Where the product can't do that, a pilot on your current work gives fairer evidence. For each task type, the report shows correctness, retries, review burden and cost.
Using pstack in Cursor · Setting up Kilo · Handing tasks to Devin
What you get from an evaluation
- A task set built from your history of merged changes, with a separate holdout set
- Results split by task type, with cost per accepted change
- Run records that let your team reproduce each result
- A brief written recommendation that also names the gaps in the evidence
What the evaluation settles
- What every accepted change costs, counting both spend and reviewer time
- Accepted changes for each tool, model and setting, counted on your codebase
- A decision on record, with the data behind it
Questions
Where does our source code go during an evaluation?
Only to the model endpoints you approve. Scope, access and each data flow are settled with you in advance.
How large does the task set have to be?
Large enough to separate the candidates. A wide gap between two of them is visible after a few tasks. A narrow gap takes more tasks and several runs of each, because an agent's output is not the same on every run. We set the size while scoping, from that variation and from the number of suitable changes in your history.
Can you separate the effect of the tool from that of the model?
Yes, as long as the tool lets you pick the model. OpenCode, a terminal agent, is one example: the model can stay constant while the tool changes, or the tool can stay while the model changes. When a tool comes with its own models and nothing else, tool and model can only be compared as one.
Can we bring other tools into the comparison?
Yes, if your organisation permits the tool and it can work from a restored checkout without access to the tests. Cline, Google Antigravity, Kiro and Devin Desktop (formerly Windsurf) are candidates. We check each one when we scope the work.
Why not rely on public benchmarks?
They are useful for narrowing a shortlist. But benchmarks such as SWE-bench take their tasks from other repositories, not yours. Replaying your team's merged work tells you how a candidate handles your code, your tests and the bar your reviewers set.
Related services
LLM cost optimisation
Agent spend broken down by team and task, then lowered through defaults, budgets and routing your platform team can run.
Harness engineering
The setup that tells an agent how to build, test and change your code, kept in version control.
Coding agent adoption
Agents taken from a few pilot teams to the rest of engineering, with training and one shared setup.
From the blog
Coding agent pilot design
How to compare coding agents on the same work, with random task assignment, a replay of merged changes and a decision rule agreed in advance.
Testing coding agent upgrades
How to check whether a new model or agent version is better on your repositories before it becomes the team default, with pinned versions and a paired comparison.
Talk to an engineer
Bring one piece of work you would like to improve. The call takes 30 minutes and needs no access to your systems or your data.
Booking runs on Google Calendar.
Prefer to write first? Leave your email and a few lines, and we reply with times.
Look for the draft in your email program
Send it from there, and an engineer replies in person with times for a 30-minute call. If no draft has opened, write to us at miki@cloudsail.com.