What 200 Pull Requests Taught Us About AI Code Review

We just crossed 200 pull requests reviewed by Revix AI. We've been tracking every comment, every confidence score, and every merge since day one, and the numbers taught us something we didn't expect about how AI code review actually works in production.

The short version: precision matters more than coverage, and the most valuable reviews happen after the code is merged, not before.

The numbers after 200 reviews

Across those 200 pull requests, Revix AI found 78 issues with over 90% confidence before they ever reached merge. Out of 216 review comments posted, 151 identified real issues that the team acted on. That's a 70% signal rate.

issues caught before merge, 90%+ confidence

78

comments that flagged real issues

151/216

signal rate across all review comments

70%

Not nitpicks. Not generic suggestions. Real issues: a query that would have missed an index, a retry loop without backoff, an endpoint that skipped the permission check every other endpoint in the module performed.

That signal rate is the metric we obsess over, because the failure mode of AI code review isn't missing bugs. It's noise.

Precision beats coverage

Every team that has tried an AI code review tool knows the pattern. The bot comments on everything. Style opinions, restating the diff, suggestions that contradict how the repo actually works. After a week, developers scroll past the comments. After a month, someone asks to turn it off.

A reviewer that comments on everything trains the team to ignore it. A reviewer that speaks up 216 times and is right 151 of them earns the opposite habit: people read the comments.

So we tuned Revix AI to stay quiet unless it has something worth saying, and to attach a confidence score when it does. The 78 highest-confidence findings were the ones most likely to become production incidents. Those are the comments that pay for the whole system.

The bigger surprise came after merge

Here's the part we didn't expect when we started.

Every merged pull request is another opportunity to learn. After code is accepted, Revix AI reads the diff again and asks a simple question:

Did this change what "correct" looks like for this repository?

Sometimes the answer is yes. The team adopted a new error-handling pattern. A module got restructured. A convention that held for a year quietly stopped applying. Merged code is the ground truth of what the team actually accepts, so it's the best training signal available.

Across 129 completed merge analyses, that feedback loop has already:

new rules created from merged code

87

existing rules refined

47

rules retired as the codebase evolved

3

Those three retired rules matter as much as the 87 new ones. They were rules that no longer reflected how the codebase had evolved. A static reviewer would have kept enforcing them, flagging correct code as wrong and eroding the team's trust one false positive at a time.

Most AI code review tools learn once

This is the gap we think most tools in this space are missing.

The typical setup looks like this: the tool scans your repository once during onboarding, builds a picture of your conventions, and starts reviewing. That picture is accurate on day one. But the repository keeps changing. New patterns get adopted, old ones get deprecated, and the reviewer's understanding stays frozen at setup time.

The result is slow drift. Six months in, the reviewer is enforcing a standard the team abandoned two quarters ago. The comments get less relevant, the false positives pile up, and the tool ends up in the same place as the noisy bots: ignored.

A code reviewer should evolve with the codebase. That's not a feature, it's the whole premise. Reviewing against a stale standard is just noise with extra confidence.

What 200 reviews add up to

We're only 200 reviews in, and we're aware that's early. But the direction of the data is clear:

• High-precision comments get read. Noisy ones train people to scroll past.
• Confidence scores let teams triage: 90%+ findings get attention first.
• Post-merge analysis is the strongest learning signal, because merged code is what the team actually accepts.
• Rules need to be retired, not just created. A growing ruleset that never prunes becomes its own source of false positives.

Watching repositories teach the reviewer over time has been the most interesting part of building Revix AI. We have a feeling this is where AI code review is headed: away from static linters with an LLM attached, toward reviewers that keep learning from every merge.

If you're new here, the launch post covers what Revix AI does end to end: PR summaries with risk levels, Conventional Comments labels, and ticket-aware reviews.

Or connect a repository and watch it start learning from your next merge: userevix.com. Every team starts with a 14 day free trial, no card required.