Live data from Hacker News

Ask HN: Teams using AI – how do you prevent it from breaking your codebase?

news.ycombinator.com

1–10 of 59 posts

Ask HN: Teams using AI – how do you prevent it from breaking your codebase?

#1
For teams actively using AI coding assistants (Copilot, Cursor, Windsurf, etc.), I'm noticing a frustrating pattern: the more complex your codebase, the more time developers spend preventing AI from breaking things.

Some common scenarios I keep running into:

* AI suggests code that completely ignores existing patterns

* It recreates components we already have

* It modifies core architecture without understanding implications

* It forgets critical context from previous conversations

* It needs constant reminders about our tech stack decisions

For engineering teams using AI tools:

1. How often do you catch AI trying to make breaking changes?

2. How much time do you spend reviewing/correcting AI suggestions?

3. What workflows have you developed to prevent AI from going off track?

Particularly interested in experiences from teams with mature codebases.

Re: Ask HN: Teams using AI – how do you prevent it from breaking your codebase?

#7
I only use those IDE-integrated assistants on small greenfield projects I work for on my own account. Sometimes they work great, sometimes they don't. (The funniest thing about Windsurf is that it keeps forgetting it's supposed to write C:\some\path on windows when it uses its tools as opposed to /C:/some/path although it will do the right thing for a short time after I advise it)

At work we have AI policies that revolve around confidentiality and a contract to use Microsoft's Copilot so that is what I do. I use it as supplement to looking up answers in the manual. For instance I had to write some complicated Mockito tests and got sample code personalized to my needs, had it explain why it did certain things, how certain things work, etc. I've also had it give me good advice about how to deal with cases that I screwed up with git.

Often it gives me the insight to confirm things in the manual quickly, but my experience is that Copilot is weak in the citation department, often it gives 100% correct answers and justifies them with 100% wrong citations.

Re: Ask HN: Teams using AI – how do you prevent it from breaking your codebase?

#8
I generally use AI as an auto complete and rarely have it write features.

In general approach it expecting to explain everything and evaluate whether it makes more sense to use it or just do it yourself. Often the second option is much faster due to muscle memory and keyboard proficiency.

Even when it comes to boilerplate it will not respect the standards (unless you throw more files in the context) so you need to be specific. In cursor you can give more files to the context in all the chats I believe (except the simple one in the current editor window) and it will do a better job.

I think too many people treat AI as a junior coder that has been exposed to the business/practices and can give him short sentences and it will understand the task, but no, it's as good as detailed is your input (which is often not worth the hassle).

In cursor you can save prompt templates in composer by the way.

That being said there are situations where LLMs can severely outperform us. An example is maintenance of legacy code.

E.g. Cursor with Claude very good at is explaining you code, the less understandable it is, the more it shines. I don't know if you've noticed but LLMs can de-obfuscate obfuscated code with ease and can make poorly written code understandable rather quick.

I've entered an 800 lines of code function that computed the final price of configurations the other day (imagine configuring a car or other items in an e-commerce) and it was impossible to understand without a very huge multi-day effort. Too many business features got glued together all in a single giant body and I was able to both refactor it (find a better name for this, document that, explain this, refactor this block to a separate function, suggest improvements and so on).

Another great use case is learning new tools/languages. Didn't use Python for a decade and I quickly setup a Jupyter Notebook for financial simulations without having to really understand the language (you can simply ask it).

AI is not limited to what we can keep in mind at the same time (between 4 or 6 informations in our short-term memory) 800 lines of context all together is nothing and you can quickly iterate over such code.

Don't misplace your expectations about LLMs, they are a tool, it makes experienced engineers shine when used for the right purpose, but you have to understand the limitations as for any other tool out there.

Re: Ask HN: Teams using AI – how do you prevent it from breaking your codebase?

#9
I've been testing LLM dev tooling and I'm still on the fence in general about whether or not the generated code is a net productivity gain for me. I still have to take the time to review the code, fix subtle bugs/corner cases and make changes to reflect the current coding standards of the existing code base. In many respects, it's like handing off a programming task to a somewhat competent intern.

Re: Ask HN: Teams using AI – how do you prevent it from breaking your codebase?

#10
I don't use AI tools quite at this level, but from the common scenarios that you list, some of these remind me of the sorts of things that teams run into when they (blindly) add new and enthusiastic developers to a project.

In those cases, one common tool to help mitigate those issues is a somewhat tedious (but very helpful) exercise of establishing a Team Charter.

I wonder if a similar sort of thing would be useful to load into the base prompt / context of every AI-generated code request? We ask our new developers to always develop with our Team Charter in mind -- why not ask Copilot to do the same?

It wouldn't address everything you listed, but I wonder if it would help.

Do you have a Team Charter and coding standards doc already written out? If not, I wonder if it could help to ask a Copilot-type tool to analyze a codebase and establish a coding standards / charter-type document first, and then back-feed that into the system.

Post reply on HN