Checking a new model or agent version on your repositories before it becomes the default

What actually changes in an upgrade

Anthropic's evaluation guidance makes a point that matters here: when you evaluate an agent, you evaluate the harness and the model together. An upgrade can change the model, the tool's version, its default settings or how it reads your instructions, and any of these can move results on your repositories.

Upgrades also arrive without anyone deciding. Claude Code's model aliases point to the recommended version and update over time, while a full model name pins a version and administrators can restrict selectable models with availableModels in managed settings. For Copilot Business and Enterprise, GitHub enables a newly designated base model automatically after 60 days, and commits to keeping a long-term support model available for a year from its designation.

Two suites with different jobs

Anthropic separates capability evaluations from regression evaluations, and the split fits an upgrade decision well.

• Regression suite: tasks your current setup passes reliably, such as typical bug fixes, test additions and dependency updates from your history. It should stay close to a full pass, so any drop is a warning.
• Capability suite: tasks your current setup struggles with, where a better model should show. It starts with a low pass rate by design.
• Behaviour checks on both: scripted checks for your constraints, such as untouched generated paths, no deleted tests and no new dependencies unless asked.

Anthropic suggests that 20 to 50 tasks drawn from real failures are a good start. Build them from merged changes, replayed in isolated copies of the repository whose history stops before the change, without later refs, remotes or network access to the merged solution. The agent sees the existing tests, and the acceptance tests added with each change are held out for grading.

Running a fair comparison

1. 1

   Hold everything else fixed

   Same tasks, instruction files, permissions and tools for both versions. Log the tool version, model ID and settings for every run, and change one of them at a time where the tool allows.

2. 2

   Start every trial clean

   A fresh environment per run, with no files, caches, agent memory or history left from earlier attempts.

3. 3

   Run several trials per task

   Results vary between runs. Report the first-try pass rate and, for tasks that must always succeed, how often all trials pass. At a 75 percent per-trial success rate, three independent trials all pass only about 42 percent of the time, and real trials are rarely fully independent.

4. 4

   Check broken tasks first

   A task that no version ever passes is, according to Anthropic, more often a broken task than an incapable agent. Fix it before it distorts the comparison.

Reading the results case by case

Compare versions per task, not only as totals: which tasks the candidate newly passes, which it newly fails and which are unchanged. A higher total can hide a regression in one task type. Then read the transcripts of every new failure. Anthropic's teams do not take scores at face value until someone has read transcripts, because a failure can come from the task, the grader or the harness as easily as from the model.

A hypothetical result: on 40 regression tasks with three trials each, the candidate keeps 37 and newly fails 3, all database migrations. On the capability suite it newly passes 6 of 25 tasks, and cost per passing task is about the same. Transcripts show it rewrites migrations that have already merged, which the instruction file forbids in a line written around the old model's habits, the kind of entry Claude Code's documentation suggests revisiting after major model releases. The team clarifies that instruction, which changes the harness, so it reruns the full regression, capability and behaviour suites with that final configuration, and runs the current version under the same instructions to see how much of the gain belongs to the model. Only then does the candidate become the default, with the old version still pinned for rollback.

From result to new default

• Agree before the run what a meaningful difference is and what would block the upgrade, and accept inconclusive as a possible outcome.
• Move an opt-in group first, then the default, keeping the previous version pinned and allowed for rollback.
• Record the decision with the suite revision, tool version, model ID and settings, so the next upgrade starts from a known baseline.
• Keep the suite alive: add tasks from new failures, move capability tasks that now pass reliably into the regression suite, and replace tasks the repository has outgrown.

Evaluations on your repositoriesDesigning a fair coding agent pilot