Au contraire - LLMs are quite bad at large scale pattern fidelity. They'll even forget key details and constraints unless told over and over again. That's why AI-written code has the quality of a patch-on-patch-on-patch.
AI changes the economics of software rewrites
21–30 of 113 posts
Re: AI changes the economics of software rewrites
#22It also changes the economics of buy vs build.
That's very true. People put up with the many limitations of off the shelf software because it's cheaper, not because it's better. Developing bespoke software solutions is now a lot cheaper than it used to be. So, there are a lot of cases where that now becomes the better option.
Doing in days what used to take months, is a bit of a game changer. Like with past cost reductions, people will underestimate the work and get it wrong. It helps if you know what you are doing rather than just vibe coding things.
But for rewrites, the sunk cost fallacy becomes a lot cheaper. So, that changes how you deal with stuff that clearly isn't living up to expectations. Unceremoniously replacing what wasn't that expensive to begin with might be the cheaper option relative to fixing it.
Re: AI changes the economics of software rewrites
#23This certainly does. If we think from this angle, it really begs the question of what language/tech stack to use if a company wants to start a new project. On one hand, if company uses a very well tech stack, development and rewrites will be faster due to AI having way more examples to draw from. In certain cases, AI will handle some edge cases which are difficult to come by/replicate under strictest test procedures.…
I don't think that holds. Internal docs for bespoke frameworks, with examples, are effective at steering AI. The main thing is that both the API and the docs are well written. Easier said than done, but you can ask AI how to write effective documentation for AI.
Re: AI changes the economics of software rewrites
#24Au contraire - LLMs are quite bad at large scale pattern fidelity. They'll even forget key details and constraints unless told over and over again. That's why AI-written code has the quality of a patch-on-patch-on-patch.
That's also true for humans.
Re: AI changes the economics of software rewrites
#25Au contraire - LLMs are quite bad at large scale pattern fidelity. They'll even forget key details and constraints unless told over and over again. That's why AI-written code has the quality of a patch-on-patch-on-patch.
Fully agree. I tried to refactor parts of a large code base with Fable+ultracode and it just keeps accidentally merging distinct concepts and making up explanations/reasonings that the code base did not contain.
For example, the code base contains a physical controller. It’s closed loop in that it can react in realtime to changes. But it’s a slightly untypical implementation because this one can even look into the future through simulations. But Fable does not understand that. Instead, I need to remind it every 30 minutes that this is closed loop. It keeps wrongly claiming that the controller was open loop and then based upon that it will make up constraints that don’t actually exist.
Re: AI changes the economics of software rewrites
#26Re: AI changes the economics of software rewrites
#27Au contraire - LLMs are quite bad at large scale pattern fidelity. They'll even forget key details and constraints unless told over and over again. That's why AI-written code has the quality of a patch-on-patch-on-patch.
That's also true for humans.
Humans will typically learn after you have forced them to apologise for the same mistake for 20 times in a row.
AI won’t.
Re: AI changes the economics of software rewrites
#28Somehow this article doesn't even mention the fact that AI makes software rewrites much, much faster than before and with higher confidence of backwards compatibility. Nowadays, a good AI harness can fairly reliably rewrite a medium complexity piece of software to an appropriate modern tech stack with pretty strong confidence of exactly preserving its behavior. The AI can pick up legacy details and keep them exactly…
AI > do a lot of things, but does it actually do that without an exhaustive test suite (which legacy software generally doesn't have, and it can never be 100%, anyway)?
Between context collapse and hallucinations, how likely is it that the end result isn't slightly polished slop that misses lots of crucial details?
Re: AI changes the economics of software rewrites
#29What's the point of the rewrite if it doesn't fix the underlying issues, though? A rewrite being a good idea often hinges on the ability to simplify. After a decade or more, it's now apparent what the application should and shouldn't do, so one can build it with those learnings and shed all tech debt from how it grew organically. Aka preserving all behavior is not what I would want from a rewrite. The point would be…
I work on a codebase from the early 2000s, a lot of it using webforms, a long abandoned .NET technology. A rewrite preserving all behavior and making no observable changes whatsoever would be amazing. But it’s also tested exactly as well as you’d expect from something like that so I’d rather not let AI go wild.
Re: AI changes the economics of software rewrites
#30Au contraire - LLMs are quite bad at large scale pattern fidelity. They'll even forget key details and constraints unless told over and over again. That's why AI-written code has the quality of a patch-on-patch-on-patch.
"It is not LLMs fault but you not knowing how to write a prompt". I know I know. But just giving all codebase and saying "rewrite it" is a no go. If e.g. going one class after class LLM will be exceptionally good at keeping the patterns and logics.
I mean it is a tool and you need to understand how the tool works. When there is too little context, where there is so much context so that you are poisoning it, when you are allowing the tool to do patch-on-patch and etc.