LLM cost optimization for coding agents
We help you work out where your coding agent and LLM spend goes, then implement the changes that hold up when tested against quality.
Engineering consulting
Coding agent evaluations on your repositories
Public benchmarks say little about how an agent performs on your code. We help you evaluate models and harness changes on tasks from your own repositories.
Discuss your setupReplaying your history
We take merged changes from your repositories, restore the code to its earlier state, and give each candidate the same task. Tests from the real change are held outside the agent's workspace, so the agent can't see or edit them.
- Same tasks for every candidate
- Cost per successful or accepted task
- Retries, rework, reverts and review findings
Limits of the evidence
Replay is limited evidence. Past tasks may not match future work, and passing tests doesn't prove a change is correct. We keep a holdout set that isn't used for tuning, and we validate on live work before anyone relies on a result.
Public benchmarks are useful for a first shortlist. They don't reflect your build system, conventions or review standards.
Using results for release decisions
Evaluations can help decide whether a new model or harness change is ready for wider use. We evaluate one change at a time and record results so your team can rerun them. This is an evolving method and doesn't replace independent validation.
Questions
Does our source code leave our environment?
Only as agreed with you. Scope, access, data flows and model endpoints are set before any work starts.
How many tasks do we need?
It depends on the question. We agree it with you during scoping.
Related services
Harness engineering for coding agents
A coding agent runs inside a harness: the instructions, commands, permissions and checks around it in your repositories. We engineer that software runtime so the agent works with your build, tests and review process.
Coding agent adoption for engineering teams
For teams adopting coding agents, whether you start with one tool in one team or bring several tools and teams together, and want a rollout you can operate yourselves.