Yes, this is actually quite common. What I've seen happen, twice personally, is a large company asking teams of developers to work on a codebase they can't really understand, and have no chance ever to, because the code has basically got to the point of no return, in terms of messiness.
What happens next IME is that progress mostly stops, and the company loses market share rapidly.
Ask HN: Have you ever worked on a product that was killed by technical debt?
61–70 of 331 posts
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#62Yes and no. The project wasn't killed specifically because "you have technical debt". It was killed because there was no way for anyone to be effective with the combination of poor undocumented code. "We need to change the email message that goes out when someone registers". This took a team of (4?) people 5 calendar days to change. As a contractor, I had to vpn in to one system, then remote desktop over another vpn…
I've also got experience of this kind of situation (See my other comment)
I think you can definitely call that technical debt.
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#63Knight Capital lost $465 million in 45 minutes caused (at least in part) by technical debt and poor development practices. Summary: http://pythonsweetness.tumblr.com/post/64740079543/how-to-lo...
> During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Was the issue technical debt or a sloppy deployment?
Which was probably due to technical debt. I can't think of another reason you'd manually copy code to 8 servers...
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#64ITA software is a good example of a company that succeeded due to the collective technical debt across their competitors. Though they only really succeeded on the shopping part. They didn't ever get to a credible booking engine that anyone would buy. Which may point to something other than tech debt being the biggest barrier to modernizing an airline reservation system.
Former ITA engineer here. Our airfare search product QPX was untouchable at the time due to design: it got results that were far better than those of the competitors because ITA was modeling the problem better (search through a graph). While competitor tech debt didn't hurt us, I don't think it was the pivotal factor in ITA's success. As you point out, our hopes of replacing a major carrier's reservation never came to fruition, unfortunately. A res system is a complex beast.
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#65It happened to me twice. The first time was in a start-up at the beginning of the century, we were developing an electronic health record and we had outsourced the database abstraction layer to a company in Greece. In the beginning things went fine but after a while the development of the DAL went slower and slower and it became unstable as well. Eventually the word came out: the main developer of the DAL framework h…
For the second one, you must have been receiving a lot of traffic for template rendering to be such a bottleneck. Why not upgrade the server?
They upgraded the server of course, to as much as they could afford. But it wasn't enough, the rendering load soon caught up. First of all because their number of visitors grew, but also because they wanted to add new features to their JSF pages and every new feature required extra rendering power as well.
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#66The closest I've come was a Rails project I inherited from a star developer who had just left the company. It was a B2B project that involved importing large Excel spreadsheets of various different formats into a standardized database for itemized review.
The code was pretty sloppy, but didn't deviate much from standard Rails idioms. Not many people on the team understood Rails well enough to read it, but I did. Bug reports were constantly flooding in. I suggested taking a sprint to build up an integration test suite and then letting loose on the backlog.
We did build up a sufficient test suite in one sprint. But the bug reports never slowed. By the time we had the confidence to truly start tackling bugs at speed, the battle had been lost. We had been so busy writing tests that we forgot to manage the bug tracker. The impression was that we were overwhelmed and unable to make progress. The project was swiftly closed.
People remembered that codebase as an exemplar of sloppy code and technical debt, but that's not the lesson I took from it. I had seen, and others would see later, much worse. The lesson I took was that perceptions are as important to manage as results.
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#67I did work on a project that was Rails 3, and for various reasons, it can never be upgraded without basically rebuilding the entire system. They keep hoping for an exit that will simply never happen because who would buy a rotting system like that? I think myself and only a few other devs there understood the situation. Its not like sales or the CEO fully understood.
You'd be surprised, most of the time for consumer apps the value isn't in the tech anyways.
This is true: the firm I work for at the moment just completed buying an incredibly expensive Fortran-based platform. That's despite the fact it cannot integrate with our existing products without serious work, nobody knows Fortran here, and the original developer sold it so he could retire.
So even though it makes no technical sense it bolsters a gap in the product offering, and they'll have to find consultants to limp it along every time they need something small done that would otherwise be very cheap. It's all about the balancing act.
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#68ITA software is a good example of a company that succeeded due to the collective technical debt across their competitors. Though they only really succeeded on the shopping part. They didn't ever get to a credible booking engine that anyone would buy. Which may point to something other than tech debt being the biggest barrier to modernizing an airline reservation system.
Former ITA engineer here. Our airfare search product QPX was untouchable at the time due to design: it got results that were far better than those of the competitors because ITA was modeling the problem better (search through a graph). While competitor tech debt didn't hurt us, I don't think it was the pivotal factor in ITA's success. As you point out, our hopes of replacing a major carrier's reservation never came t…
I do agree that QPX was untouchable, but I still think tech debt in competitors was a major factor. There were plenty of smart people at your competitors...I'm sure graph search occured to them. I suspect efforts to green field that were squashed...nobody wanted to throw out the hairball they had because of the existing investment. Thus, they tried to "fix" what they already had...with obviously bad results.
Edit: And, worth mentioning that your competitors wouldn't have had to be better than, or even as good as QPX. "Good enough" would have squashed several big sales, since shopping was typically bundled in with what their customers already paid.
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#69[deleted]
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#70Yes and no. The project wasn't killed specifically because "you have technical debt". It was killed because there was no way for anyone to be effective with the combination of poor undocumented code. "We need to change the email message that goes out when someone registers". This took a team of (4?) people 5 calendar days to change. As a contractor, I had to vpn in to one system, then remote desktop over another vpn…
This has been my experience. Since technical debt is hard to measure, it's more a case of a series of unwise technical decisions leading to a lack of productivity. Due to tight schedules, short-cuts are taken which lead to more unwise technical decisions, and you have a death-spiral.
Isn't this precisely technical debt? Unless you want to split hairs and call this a technical massacre...