Very vague post light on details, and as usual, feels more like a marketing pitch for the website.
I found it useful.
Learnings from building AI agents
11–20 of 70 posts
Re: Learnings from building AI agents
#12Re: Learnings from building AI agents
#13what I saw using 5-6 tools like this: - PR description is never useful they barely summarize the file changes - 90% of comments are wrong or irrelevant wether it's because it's missing context, missing tribal knowledge, missing code quality rules or wrongly interpret the code change - 5-10% of the time it actually spots something Not entirely sure it's worth the noise
code-reviews are not a good use-case for LLMs. here's why: LLMs shine in usecases when their output is not evaluated on accuracy - for example, recommendations, semantic-search, sample snippets, images of people riding horses etc. code-reviews require accuracy.
What is a useful agent in the context of code-reviews in a large codebase is a semantic search agent which adds a comment containing related issues or PRs from the past for more context to human reviewers. This is a recommendation and is not rated on accuracy.
Re: Learnings from building AI agents
#14> Encouraged structured thinking by forcing the AI to justify its findings first, significantly reducing arbitrary conclusions.
Ah yes, because we know very well that the current generation of AI models reasons and draws conclusions based on logic and understanding... This is the true face palm.
Re: Learnings from building AI agents
#15[flagged]
Re: Learnings from building AI agents
#16Very vague post light on details, and as usual, feels more like a marketing pitch for the website.
It's recreating the monolith vs micro-service argument by proxy for a new generation to plan conference talks around.
Re: Learnings from building AI agents
#17I agree with the sentiment of this post. I my personal experience the usefulness of a LLM positively correlated with your ability to constrain the problem it should solve.
Prompts like 'Update this regex to match this new pattern' generally give better results than 'Fix this routing error in my server'.
Although this pattern seems true empirically, I've never seen any hard data to confirm this property(?). And this post is interesting but seems like a missed opportunity to back this idea with some numbers.
Re: Learnings from building AI agents
#18I think they skipped over a non-obvious motivating example too fast. On first glance, commenting out your CI test suite would be very bad to sneak into a random PR, and that review note might be justified.
I could imagine the situation might actually be more nuanced (e.g. adding new tests and some of them are commented out), but there isn't enough context to really determine that, and even in that case, it can be worth asking about commented out code in case the author left it that way by accident.
Aren't there plenty of more obvious nitpicks to highlight? A great nitpick example would be one where the model will also ask to reverse the resolution. E.g.
final var items = List.copyOf(...);
Re: Learnings from building AI agents
#19Lessons.
[flagged]
Re: Learnings from building AI agents
#20"After extensive trial-and-error..."
IMO, this is the difference between building deterministic software and non-deterministic software (like an AI agent). It often boils down to randomly making tweaks and evaluating the outcome of those tweaks.