GitHub open sources ReviewBench for AI code review agents

GitHub published ReviewBench, a benchmark for code review agents, built on representative GitHub pull requests with multi source ground truth, calibrated evaluation and production aligned metrics. The post, by Michelle Zhou and Alejandro Carderera de Diego, opens on a practical problem: agentic code review is becoming a normal part of how development happens, but the quality of existing AI reviewers is hard to measure, and some reviewers surface more issues while others produce more noise, and some catch critical problems where others only pick at small improvements.

The argument for a new benchmark is that a good one has to reflect the diversity of real pull requests, cover a broad set of review findings, and support breakdowns by severity, category and by precision recall preferences, since teams need code review to do different things in different workflows. The existing landscape, GitHub says, forces tradeoffs between label quality, coverage and how closely the data mirrors real review, which leaves a gap for a rigorous and reproducible evaluation methodology.

The intended audience is teams building code review agents, and the specific promise is an offline signal that reliably tracks whether a change is likely to improve the production experience, so a developer can evaluate a reviewer’s strengths before adopting it rather than discovering them in a live pull request. GitHub states the benchmark is available to use today.

(Original source: GitHub)