Measuring a GitHub Copilot pilot beyond suggestion counts

What GitHub's Copilot metrics measure

GitHub groups Copilot usage metrics into adoption, engagement, acceptance rate, lines of code and pull request lifecycle. They come through a dashboard with 28-day trends, an impact dashboard, a usage metrics API with organisation, repository and user reports, and NDJSON exports. Data for a day is available within two full UTC days.

Most of it comes from client-side IDE telemetry, so the richest data needs telemetry enabled and a supported IDE and extension version. Server-side telemetry still counts users whose client telemetry does not arrive, so active-user totals hold up, but their feature and lines-of-code breakdowns stay empty. Copilot Chat on GitHub.com and GitHub Mobile are not included.

Layer one: did the pilot actually happen?

Usage data answers whether the tool was used enough to judge it. GitHub's interpretation guide calls a weekly-active-to-license ratio above 60 percent healthy ongoing usage. The impact dashboard sorts users into cohorts: passive, code first, agent first and multi-agent, based on which surfaces they used on at least two days in the trailing 28.

Use this layer as a precondition, not as the verdict, and read the cohorts carefully. Passive does not mean idle: GitHub notes that someone who asks Copilot Chat or agent mode questions without applying code edits can stay in the passive cohort. A large passive share is a reason to find out how the group actually works with Copilot, through licences, IDE setup and a short survey, before you read anything else.

Layer two: what happens to pull requests

The usage metrics API reports daily pull request activity, including pull requests created by Copilot cloud agent, how many of them merged, their median time to merge, and Copilot code review suggestions and how many were applied, broken down by comment type. Repository-level reports add review stage durations for pull requests that people opened and other people reviewed.

The daily counts give throughput: Copilot pull requests created and merged per day. Creations and merges are separate events on different dates, so merges divided by creations for the same week is not a merge rate and can exceed 100 percent. For a merge rate, follow a fixed set of Copilot pull requests by ID from creation through a stated window, such as 14 days, using your git host's data. Review effort is another gap: GitHub's review stage durations exclude pull requests created by bots, Copilot cloud agent included, and those merged without a qualifying human review, so measure review time on agent pull requests from your git host's events too.

Layer three: the data GitHub does not have

Whether merged work held up is in your own history. Copilot cloud agent's commits are authored by Copilot with the requesting developer as co-author, which makes its changes easy to label. From that label, track per team and repository:

• Rework: follow-up commits that fix a merged change within an agreed window
• Reverts, and Copilot pull requests closed without merging
• Review time and rounds per Copilot pull request, from your git host's events, since GitHub's stage durations leave bot-created pull requests out
• Spend from your billing data, divided by merged changes that needed no rework

Where the numbers mislead

• Acceptance rate and lines of code count output. GitHub calls lines of code a directional measure, and agent edits do not follow a suggest-then-accept flow.
• The adoption multiplier compares different groups of users. GitHub itself says team composition and seniority can influence it as much as adoption, and that its return on investment view shows associations, not causes.
• Team-level reports exclude teams with fewer than five seated Copilot users, so team rows add up to less than the organisation total.
• Organisation-level metrics count a user in every organisation they belong to, so organisation figures cannot be summed.

A decision sheet for continuing

The thresholds below are hypothetical and should be replaced with your own, agreed before the pilot starts and read per team and task type:

• Expand: active usage is steady across the pilot group, the cohort merge rate of Copilot pull requests meets a level you set in advance, and rework, reverts and review time are no higher than the baseline.
• Hold and fix: usage is healthy but rework or review rounds rose. Improve repository instructions, setup steps and task writing, then measure again.
• Extend: too little usage or too few merged pull requests to judge. The pilot has not answered the question yet.
• Stop: usage stayed low after enablement, or review burden rose without more work merging.

Designing a fair coding agent pilotGitHub Copilot implementation