31–40 of 59 posts
It sounds like you're not training it with your existing code base, and that you're running it with relatively small contexts. Have you done any custom LLM training on your code base, and what model are you using?
Who is training models on a code base? That's an extraordinary use case.
1. We never catch AI trying to make breaking changes, but we catch developers who do. Since using AI tools we haven’t seen a huge change in those patterns.
2. Prior to opening a PR; developers are now spending more time reviewing code instead of writing it. During the code review process, we use AI to highlight potential issues faster.
3. Human in the middle
The latter problem should be prevent by code review (first by the developer using the AI tool and then their teammates on a PR.) Code generated by AI should be reviewed no differently than code written by a human. If you wouldn't approve the PR if a person wrote this code, why would you approve it because an LLM wrote it? If your PR process is not catching these issues, you have a PR process problem, not an AI problem.
The former problem I have no idea.
2. Use LLM for code auto complete or ask LLM to code specific functions that you weave together to deliver a feature.
3. Or use it to explain your code. I concatenate all my code and shove it into Gemini and ask it to explain how the legacy stuff works
The objective of my solver was to get good solutions using only RAG (no embeddings) and with minimal cost (low token count).
Three techniques, combined, yielded good results. The first was to take a TDD approach, first generating a test and then requiring the LLM to pass the test (without failing others). It can also trace the test execution to see exactly what code participates in the feature.
The second technique was to separate “planning” from “coding”. The planner is freed from implementation details, and can worry more about figuring out which files to change, following existing code conventions, not duplicating code, etc. In the coding phase, the LLM is working from a predefined plan, and has little freedom to deviate. It just needs to create a working, lint-free implementation.
The third technique was a gentle pressure on the solver to make small changes in a minimum number of files (ideally, one).
AI coding tools today generally don’t incorporate any of this. They don’t favor TDD, they don’t have a bias towards making minimal changes, and they don’t work from a pre-approved design.
Good human developers do these things, and this is a pretty wide gap between adept human coders and AI.
Because if it is, the answer seems fairly simple: you install a proper code review process, and you don't allow shit code into your codebase.
These are all problems with regular humans, too. You prevent it via code reviews.
Do you not have code reviews? These are all problems with regular humans, too. You prevent it via code reviews.