Choosing a coding agent or AI consultancy: the questions that matter

Ask how they will measure where you start

A consultancy that promises a saving before it has seen your usage data is guessing. Ask what it will measure in the first weeks and from which data. For coding agents, useful starting measures are the cost per merged change, the share of agent changes that are accepted, and how much review and rework those changes need. For process agents, they are the cost and the accuracy per handled case.

Evidence to ask for: an example of a baseline from an earlier engagement, with the client's name removed if needed, and the list of exports or access the consultancy would need from you.

Ask for a test on your own work

Demonstrations on sample repositories show what a tool can do, not what it will do in your codebase. A serious consultancy proposes to test changes on work your teams have already merged, or on cases your process has already handled, and agrees before the test what counts as success.

If it compares tools, ask how it keeps the comparison fair. Giving one tool to one team and another tool to a different team mostly measures the difference between the teams.

Designing a fair coding agent pilotEvaluations on your repositories

Ask who will do the work

Ask for the names of the engineers who would work with your team, and what they have built or run with Claude Code, Cursor, GitHub Copilot or OpenAI Codex. Ask whether they will work in your repositories and reviews, or hand over a report and leave.

Vendor partner programmes help, but read them for what they measure. Anthropic's services partner tiers count certified practitioners, joint customers in production and public customer stories. OpenAI launched its own partner network in June 2026. A tier shows that a firm has invested in a vendor. It does not show results on your code.

Ask what stays with your team

Ask what your team will own when the engagement ends: the instruction files and checks for the agents in version control, the evaluation sets, the runbooks, and the knowledge to change all of it without the consultancy. If the answer depends on a proprietary platform you cannot see into, you are buying a dependency.

Ask too whether the consultancy is tied to one tool vendor. That can be fine if you have already chosen the tool. If you have not, an independent comparison on your own work is worth more than a reseller's recommendation.

Ask about access, data and your works council

A first step rarely needs production data. Ask whether the consultancy can start from usage and billing exports and merged changes, which systems it would need to read, and where any data would be processed.

In Germany, introducing technical systems designed to monitor employees' behaviour or performance needs the works council's agreement under section 87(1) no. 6 of the Works Constitution Act. A consultancy that measures teams, tools and tasks rather than individual engineers makes that question simpler, and that often decides whether the first step starts this month or next quarter.

Ask for a small first step with a written result

A good first engagement is short, has a fixed scope and price agreed in writing, and ends with a result your team keeps whether or not you continue. Two to four weeks with one team is usually enough to measure a baseline, change something and measure again. If a consultancy can only start with a long programme, ask what you would know after the first month.

Our two-week cost review for one team

Questions for the first call

• What will you measure in the first two weeks, and from which data?
• How will you test changes on work we have already done?
• Who exactly will work with our team, and what have they shipped?
• What will we own at the end, and can we change it without you?
• Which access do you need, and where is our data processed?
• What does the first step cost, and what will we know when it ends?