CodeRabbit Blog

At the time of writing, the first sample image on that page is this:

https://coderabbit.ai/assets/section-1-f9a48066.png

which recommends adding a "maxIterations" counter to the "for len(executedComponents) ..." loop here:

https://github.com/fluxninja/aperture/blob/26e00ea818c7c28dac00c1a5e1a02e555589ec7a/pkg/policies/controlplane/runtime/circuit.go#L274-L280

HOWEVER

- the review has failed to notice the logic using "numExecutedBefore" (around line 377) that already prevents the specific bug it is suggesting a fix for

- the suggested change decrements "maxIterations" inside the "for ... range circuit.components
{" loop which means it isn't counting iterations, it's counting components

This kind of suggestion is particularly nasty because it's unlikely that the test suite populates enough components to hit "maxIterations" - so an inattentive reader could accept it, get a green build, and then deploy a production bug!

Sorry for the late response!

My guess is they didn't publish it because they only measured it at one company, if they had the data across the cohort they would have published.

The general result that review/re-wrok can cancel out the productivity gains is supported by other studies

AI generated code is 1.7x more buggy vs human generated code:
https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report

Individual dev productivity gains are offset by peers having to review the verbose (and buggy) AI code:
https://www.faros.ai/blog/ai-software-engineering

On agentic being the saviour for productivity, Meta measured a 6-12% productivity boost from agents programming:
https://www.youtube.com/watch?v=1OzxYK2-qsI&si=ABTk-2RZM-leTxtR

"But it's different now" :)

> Where are the "code-review" agents at?

OpenAI's Codex Cloud just added a new feature for code review, and their new GPT-5-Codex model has been specifically trained for code review: https://openai.com/index/introducing-upgrades-to-codex/

Gemini and Claude both have code review features that work via GitHub Actions: https://developers.google.com/gemini-code-assist/docs/review-github-code and https://docs.claude.com/en/docs/claude-code/github-actions

GitHub have their own version of this pattern too: https://github.blog/changelog/2025-04-04-copilot-code-review-now-generally-available/

There are also a whole lot of dedicated code review startups like https://coderabbit.ai/ and https://www.greptile.com/ and https://www.qodo.ai/products/qodo-merge/

I use CodeRabbit that helps, but it does not fix the two root issues. I run their free VS code plugin to review local commits first, which catches nits, generates summaries, and keeps me in my editor. The PR bot then adds structure so humans focus on design and invariants. Review state still lives in the forge, not in Git, and interdiffs still depend on history. If Git gets a stable Change-Id, storing review metadata in Git becomes realistic. Until then this is a pragmatic upgrade that reduces friction without changing the fundamental.

https://www.coderabbit.ai/ide

https://www.coderabbit.ai/blog/our-response-to-the-january-2025-kudelski-security-vulnerability-disclosure-action-and-continuous-improvement

> No customer data was accessed

As far as I can tell this is a lie.

The real answer is that they have absolutely no clue if customer data was accessed, and no way to tell. I'm not even sure Github could tell, but it's not clear if the exploits way of generating private keys to access private repositories is any different to what CodeRabbit does in normal operation.

Interesting use-case: We recently started using ast-grep at CodeRabbit[0] to review pull request changes with AI.

We use gpt-4 to generate ast-grep patterns to deep-dive and verify pull-request integrity. We just rolled this feature out 3 days back and are getting excellent results!

Comments such as these are powered by AI-generate ast-grep queries: https://github.com/amorphie/contract/pull/100#discussion_r1420655994

[0]: https://coderabbit.ai