The company's entire code base was split into two: Java from the 'enterprise' stuff, and Perl for everything else. The Java side had something close to 20 million lines of code, and the Perl side had about one million lines of code. As always, I worked on the operational side.
NB: I rather like Perl, though I haven't used it much at work since leaving this company a few years ago. The ...things that follow are not an indictment against Perl. The problem wasn't the language.
Even used properly, Perl is a pretty dense language, so a million lines of Perl in one place is quite impressive.
The company's code started in the mid 90s and continued to grow. When I arrived, there were a couple of guys on the operations tools team with me that had been there 7-9 years I believe. They understood the code better than anybody, and they were both very bright guys, but things came up, at least a couple of times a week, that would surprise and sometimes mystify them. After some digging, debugging and sniffing around, they'd usually approximately get somewhat near a root understanding.
That's fairly amazing to me. I've helped create some enormous code bases my decades long career, in one case, even larger than a million lines. And I'd certainly find myself surprised from time to time, but never for very long.
So these millions of lines of code had, over the prior 17 years, grown organically, and had, essentially, never been refactored.
The code itself was, for the most part, quite tidy. And the underlying concepts and structures were pretty simple and elegant. They represented approaches I mostly agreed with.
But...almost nothing had ever been removed. No refactors, over millions of lines of code, over most of 20 years.
There were side effects everywhere. Many of the deep, underlying methods had had so many arguments added that perhaps one third them would be useful or used in any given invocation.
And, of course, there were virtually no tests.
After somewhat getting up to speed on this system, I declared it DOA, and started to push for a complete rewrite. I've been around the block enough times to understand and comprehend the hazards of that approach, and I didn't mentally pull that trigger lightly or quickly.
But....the organization would not have it. The two senior guys on my team were fine with the idea, but many other long-timers were not.
That code is, to this day, with no doubt another 100k lines of Perl added to it, (poorly) powering fundamental and important pieces of a huge company you have all heard of.
So management basically threw bodies at it. Lots of bodies. And, even though the operational quality of the products were objectively fairly poor, the particular market niche didn't need or demand better, so the company made and continues to make a ton of money.