51–60 of 71 posts
Does anyone have an open PR on a public repo? I'll run this against your PR for you with my CC credits as a sort-of benchmark! Send me your PR link :) I'm going to create one on one of my other repos meanwhile and add a link to the review when it's ready.
The best code review improvement I have done in my workflow with Claude is using tuicr ( https://tuicr.dev ). It runs locally, YOU review all the code locally, and feedback that to Claude. Agents reviewing AI code always felt dirty to me, especially when working on production (non-disposable) code.
As in, if you have a large screen, a particularly-trained/prompted AI can organize the code changes in a "flowchart" with floating windows you can easily follow
Maybe in this UI, each code piece also comes with a summary from an agent that has already auto-reviewed the whole PR and creates a basic summary (instructed to be neutral but surface issues if it finds any).
Earlier quoted context omitted.
That's a great idea. I had trouble finding anything like this, a benchmark made for (AI) code reviewers. I had expected to find something like an eval harness available on GitHub, but couldn't find it. Any suggestions? Or maybe we/I/someone should build something like this? I suppose one challenge is that if it's going to be publicly available, it would also be easy to cheat, but still seems it would be useful if peo…
https://www.codereviewbench.com/ https://codereview.withmartian.com/
Earlier quoted context omitted.
How do you prevent it from increasing scope? That's the main issue I've found from running loops like this. Each loop has ~7 agents, say, looking through different lenses (security, UX, performance, etc.). Each one notes a few issues, each issue gets fixed, you do 5 to 8 loops, as you say. Each individual item that gets fixed looks minor but when you add it all up at the end you've increased PR size and scope signifi…
That is such a good point. I recently opened a PR against this AI personal finance tool Ray https://github.com/cdinnison/ray-finance/pull/8 to add an Apple Card import feature, since Apple Card is not supported by Plaid. I built the manual import feature, opened the PR, and then ran a code review. What I hadn't thought about when I built the feature, was the myriad ways that the implications of importing data from Ap…
- one wave is code reduction via DRY removals and architectural fixes, and another is adverserial to get rid of false additions, so this helps AI bloat either way
- as the other comment says, underspecification is a problem, so this ends up finding when the implementation, tests, docs, quality guide, and spec are out of sync, with whichever to blame.
- Usable, well-designed, secure, and well-typed code ends up being bigger, so this helps cut to the chase. Ultimately, either you get there or you don't, and this helps cut review burden so you can do your part of it faster and at a higher level.
Funny enough, I'm now playing with gardening agents whose job it is to reduce code. But I wouldn't want to slow PRs on that so view as seperate PRs.
> Runs against your regular Claude Code subscription (Max plan recommended) — unlike /ultrareview, which charges against your Extra Usage pool. How expensive is it to run in your experience? In $ or tokens?
On small PRs (small features / changes ~hundreds to thousands of lines), I'd say around 500,000 total tokens. On large PRs (new feature sets for apps ~10,000-30,000 lines), around 2-3 million total tokens. By the way, I should have mentioned in my original post, adamsreview counts tokens used by sub-agents across the stages, and tells you at the end of each stage the total used so far.
Great project! I’ve build something similar, not very clean and polished, but focussed around deterministic orchestration of multiple agents via typescript, because a coordinating agent was notoriously bad at things such as fetching relevant tickets and other context. One thing I struggle with so far, though, are the actual instructions for the review themselves. They are either too vague, leading to superficial or o…
That's awesome to hear and I'd love to see it when you're ready. I actually think having something like adamsreview orchestrated by deterministic code - instead of simply having AI agents use deterministic code occasionally as this app does - could be even better! The problem I ran into is that if you build a deterministic app that happens to use LLMs instead of the other way around, I don't think there's any way to…
Why not just use an eval harness to prove this catches more real bugs? Benchmarks on actual bug classes would be far more convincing than comparing against /review.
That's a great idea. I had trouble finding anything like this, a benchmark made for (AI) code reviewers. I had expected to find something like an eval harness available on GitHub, but couldn't find it. Any suggestions? Or maybe we/I/someone should build something like this? I suppose one challenge is that if it's going to be publicly available, it would also be easy to cheat, but still seems it would be useful if peo…