When a coding agent's change passes locally and fails in CI

Three places the difference comes from

An agent's local environment may be a hosted sandbox, a CI-like runner or a container on someone's machine, each assembled by its own setup. That gives three distinct sources of a pass that turns into a failure.

• The environment differs: runtime or tool versions, a dependency installed without the lockfile, a database or queue that runs in one place only, network rules, or credentials available in one environment and not the other.
• The commands differ: the agent ran one test file, skipped lint or type checks, or followed an instruction file that names a different command from the CI workflow.
• The test is nondeterministic: it depends on order, time, parallel execution or shared state, and passes or fails on the same commit. The instability may predate the change or be introduced by it.

A procedure that narrows it down

1. 1

   Read what the agent actually ran

   Open the session log, not the agent's summary. Note the exact commands, their exit codes and whether the full suite ran. Anthropic's evaluation guidance distinguishes what an agent says it did from the outcome in the environment, and the same distinction applies here.

2. 2

   Rerun the failed job with debug logging

   On GitHub Actions, rerun only the failed jobs with runner diagnostic and step debug logging. A rerun uses the same commit and ref as the original run, so a different result points to the environment or to nondeterminism, not to new code.

3. 3

   Compare the base and the agent's commit

   A pass on rerun does not clear the change, because a race the agent introduced also fails only sometimes. Run the failing test repeatedly on the base commit and on the agent's commit, with the same dependencies and services, and compare failure rates and error signatures. Failures only with the change point to the change. Failures on both do not clear it either, because the change can make an existing flake fail more often or differently. Classify the test, give it an owner and a ticket, and do not let the agent make it pass by loosening assertions.

4. 4

   Reproduce the CI command in a clean container

   Use the CI image and versions, run the exact CI command and compare runtime versions, installed packages, environment variables and running services with the agent's environment.

5. 5

   Check the agent's setup log

   Look for setup steps that failed and requests that were blocked. Some hosted agents keep working after a failed setup step, as described below.

gh run rerun RUN_ID --failed --debug

Symptoms and where to look first

• Module not found or a different version only in CI: the agent installed without the lockfile or used another runtime version.
• Integration tests fail in one place only: a database, queue or seed data exists in one environment and not the other.
• Lint or type errors only in CI: the agent ran tests but not the full check, or its instructions name another command.
• Network or authentication errors: a host allowed in one environment is blocked in the other, or a credential is not delivered the way the program expects.
• Results change between reruns of one commit: something is nondeterministic, in the test, the change or both. Compare repeated runs on base and head before classifying it.

Traps specific to hosted agents

GitHub's Copilot cloud agent runs a copilot-setup-steps job before it starts. If a step fails, the agent skips the remaining steps and works with the environment as it is, so a missing dependency surfaces later as a strange test result. Requests blocked by its firewall appear as a warning in the pull request with the address and the command. And on a Copilot pull request, workflows do not run until someone with write access selects Approve and run workflows, so a missing CI result is not a pass.

In OpenAI's Codex Cloud, network secrets reach programs as placeholders that a proxy replaces for allowed HTTPS destinations. A tool that reads the raw value gets the placeholder and fails in a way CI does not. Existing Codex tasks also keep their own saved files and installed tools, so a long-running task may not match a newly published environment.

Fix the harness, not only the change

A hypothetical case: CI runs tests in parallel shards, while the agent runs them in one process. A test that relies on state left by another passes for the agent and fails in shard three. Fixing that one pull request helps once. The lasting fixes are in the harness:

• One check command that the instruction file names and CI runs, with the same test ordering and parallelism
• Runtime and tool versions pinned in one place that both environments read
• Services started from the same container definition for the agent and for CI
• A rule that the pull request description lists the commands run and their results
• Tests classified as unstable after base-and-head comparison labelled, owned and tracked, and kept out of the agent's fast set until fixed

If CI keeps failing for environment reasons on one task type, move that type back to interactive work until the harness is fixed. This diagnosis and repair is part of our harness engineering work with platform teams.

Making a repository ready for coding agentsHarness engineering