Agent ROI Metrics Need Work Receipts

GitHub's 7 August changelog added a potential return on investment section to the Copilot impact dashboard. It compares Copilot cost per developer, cost as a share of payroll, and pull requests per developer across adoption cohorts, including developers who are primarily in chat and completions versus more agent-first developers.

The same release cluster made Copilot code review effort levels generally available, with Lite and Balanced review modes, organization-level defaults, and visibility into which effort level was used. GitHub also updated the Copilot usage metrics API to report agent app activity, including agent_id, agent_name, user-initiated interaction counts, and aggregated session counts for recognized third-party agent apps.

Two adjacent updates matter for the same operating story: GitHub Projects now has a public-preview Relates to issue relationship and generally available multi-select fields, while GitHub Code Quality stopped automatically adding Copilot as a reviewer. Fresh evidence is moving in two directions at once: AI work is becoming easier to measure, and the decision to invoke AI review is becoming more explicit.

The repeated angle to avoid

The last ten posts here covered prototype exit inventories, Copilot billing handovers, code-quality setup, comment-triggered agents, AI gateway budgets, model entitlements, worktree isolation, stacked pull requests, review-agent context, and package-malware controls. Older overlapping posts covered Copilot metrics, AI coding telemetry, project taxonomies, model routing, cost chargeback, pull-request dashboards, and review environments.

The weak version of this article would repeat the old X needs Y pattern: AI productivity needs measurement. That is true and too generic. The sharper thesis is that AI ROI dashboards are only useful when operators can attach each measured unit of activity to a receipt for the work it changed. A pull request is an output count. It is not proof that the right agent, effort level, model posture, and review path improved a business surface.

ROI dashboards create a useful temptation

The new Copilot impact view is directionally helpful. Most small teams are past the question of whether AI tools are being tried. They now need to know whether deeper adoption justifies the spend, which teams are underusing paid capacity, and whether agent-first workflows are producing enough useful change to deserve more enablement.

Putting AI credit consumption beside payroll assumptions and pull-request output is a better conversation than arguing from anecdotes. It gives a founder, engineering lead, or operations owner a starting point for questions like:

Dashboard signal	Useful question	Dangerous shortcut
Cost per developer	Is AI spend material compared with payroll?	Treating low cost as proof that all usage is worthwhile
Cost as payroll share	Which compensation assumptions change the business case?	Pretending salary bands explain the value of work
Pull requests per developer	Did adoption correlate with more shipped changes?	Rewarding PR volume without checking quality or surface risk
Adoption cohort	Are agent-first users getting different results?	Assuming deeper adoption is always the maturity target
Agent app activity	Which agents are actually being used?	Comparing agents without normalising task class and evidence

The mechanism is simple: ROI dashboards compress messy work into comparable numbers. That compression is useful for governance, but it drops context. A one-line documentation fix, a checkout bug repair, a security-sensitive refactor, and an abandoned agent branch can all touch the same productivity denominator if the team does not keep receipts.

Agent app metrics need stable joins, not names in a chart

The usage metrics API change is especially important because agent activity no longer has to sit in one blended bucket. GitHub says the new totals_by_3rd_party_agent array includes one entry per recognized agent app, with a stable agent_id and a display agent_name that can change. That stable identifier is the practical detail operators should not miss.

If you group by display name, your history can split when a vendor renames an app. If you group only by user, you cannot tell whether a developer used Copilot coding agent, Codex, Claude, or another partner agent for different work. If you group only by repository, you may miss that one agent is mostly drafting issues while another is changing payment-adjacent code.

For Paul's audience, the useful join is not agent_id -> cost. It is:

agent_work_receipt:
  reporting_period: 2026-08-08
  agent_id: stable_agent_identifier
  agent_name_seen: display_name_at_collection_time
  actor: github_user_or_app_identity
  repository: owner/repo
  business_surface:
    - checkout
    - lead_funnel
    - analytics
  task_class: review | diagnosis | mutation | documentation | backlog_creation
  effort_level: lite | balanced | not_applicable
  interaction_count_source: copilot_usage_metrics_api
  session_count_source: aggregated_report_when_available
  work_evidence:
    pull_request: https://github.com/example/repo/pull/123
    issue: https://github.com/example/repo/issues/456
    checks: build_passed | tests_failed | not_run
    outcome: merged | closed_unmerged | draft_only | follow_up_created
  operator_note: why_the_work_was_worth_or_not_worth_the_spend

This is intentionally small. It is not a second analytics warehouse. It is the missing bridge between a platform metric and an operating decision.

Effort levels turn review depth into cost and risk metadata

Copilot code review effort levels becoming generally available adds another field to the receipt. Lite is for straightforward changes. Balanced is for deeper analysis from a higher-reasoning model. Organization defaults can be inherited, and individual reviews can still choose the level.

That means effort level is no longer a hidden preference. It is a review policy signal.

A practical default matrix might look like this:

Change type	Default AI review posture	Human checkpoint
Copy, docs, comments, low-risk UI text	Lite	Normal reviewer confirms intent
Component refactor behind unchanged behaviour	Lite, escalate if diff crosses surfaces	Reviewer checks screenshots and tests
Quote, booking, checkout, pricing, CRM, or analytics logic	Balanced	Owner of the business surface reviews evidence
Authentication, secrets, payment, privacy, or security-sensitive code	Balanced plus explicit human approval	Security or senior engineering review required
Generated follow-up issues or backlog grooming	Lite or no AI review	Operator samples for noise and duplicates

The failure mode is not that teams choose the wrong label once. The failure mode is that the ROI dashboard later says agent-first work produced more pull requests, while nobody can tell whether those pull requests were reviewed at an appropriate depth. More output with shallow review on risky surfaces is not higher ROI. It is delayed risk.

Automatic reviewer removal is a governance clue

GitHub Code Quality no longer automatically creates a ruleset that requests Copilot review on every pull request. GitHub says the previous Code Quality-created ruleset settings for automatic Copilot review, review on new pushes, and draft PR review have been disabled where they still match the generated rule.

That is a small product reversal with a useful lesson: automatic AI review is not neutral quality infrastructure. It consumes attention, credits, and trust. It can also create review fatigue if every draft, typo fix, and generated dependency update receives the same AI reviewer ceremony.

For small web teams, this supports a better rule:

1. Default to explicit invocation for risky or meaningful changes. Do not request AI review everywhere just to feel covered.
2. Use rules where the surface justifies it. Checkout, booking, security, analytics, accessibility, and structured-data changes may deserve automatic review.
3. Keep draft behaviour deliberate. Early draft reviews can be useful for architecture, but noisy for work-in-progress copy or generated scaffolds.
4. Record the effort level used. A review comment is easier to interpret when the operator knows whether it was Lite or Balanced.
5. Sample false positives and misses. ROI should include avoided bugs and review quality, not only PR count.

The governance lesson is that AI review belongs in the same release-control conversation as branch protection, CODEOWNERS, project fields, and test requirements.

Multi-select project fields make receipts less brittle

The new Relates to issue relationship and generally available multi-select fields help because AI work rarely belongs to one tidy category. A checkout fix can also be an analytics change. A local-service quote-form update can be lead quality, spam prevention, CRM routing, accessibility, and conversion work at the same time.

If the project taxonomy forces one label, the ROI story becomes distorted. The agent that "improved frontend" may actually have touched revenue reporting. The PR that looked like a small UX change may have repaired a lead-routing problem. The high-effort review may have been justified because a multi-select field showed both payment and privacy.

Use multi-select fields to keep ROI analysis honest:

Field	Why it matters for AI ROI
business_surface	Connects activity to checkout, booking, product pages, local SEO, analytics, or internal tools
risk_class	Separates security, privacy, accessibility, revenue, reliability, and content risk
ai_role	Distinguishes draft, diagnosis, review, mutation, and backlog creation
evidence_required	Names the proof needed before counting the work as useful
outcome	Prevents abandoned agent activity from looking like shipped value

Relates to is useful for a different reason. It lets operators connect issues that share context without pretending one blocks or duplicates the other. That matters when one agent session creates follow-up work, another fixes a related bug, and a dashboard later needs to understand the cluster of work rather than count isolated tickets.

A decision rule for AI productivity reviews

Use this rule before increasing AI seats, enabling more agents, or changing default review effort:

Count an AI workflow as productive only when the metric row can be joined to a work receipt with a named surface, task class, evidence URL, review posture, and outcome.

Then review four buckets each month:

1. Valuable shipped work. Merged changes with clear business surface, passing evidence, and appropriate review depth.
2. Useful non-merged work. Diagnoses, summaries, or issue decomposition that prevented waste or clarified decisions.
3. Noisy activity. Sessions, reviews, or generated follow-ups that created attention cost without a useful artifact.
4. Risky acceleration. More output on surfaces where evidence, ownership, or review depth did not keep up.

The dashboard answers whether adoption and spend changed. The receipt answers whether the change was worth trusting.

The practical takeaway

GitHub's fresh ROI, agent metrics, review-effort, project-field, and Code Quality updates are not just Copilot administration. They are pieces of an operating system for AI-assisted work.

For a small business or web team, the next maturity step is not a bigger AI dashboard. It is a thinner, more reliable join between AI activity and business evidence. Keep the stable agent identifier. Preserve the effort level. Label the business surface with multi-select fields. Connect related work without forcing fake dependencies. Count pull requests only after the receipt shows what changed, why it mattered, and how it was reviewed.