Live data from Hacker News

Ask HN: Have you ever worked on a product that was killed by technical debt?

news.ycombinator.com

231–240 of 331 posts

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#231
post #109

There are multiple cases of "killed by technical debt". There's the case of mysterious and unsolvable breakage. The product simply stops working, and the team is unable to get it working again, period. This can happen with really ancient legacy products where the original team is gone, or young products that are written badly by inadequate teams. There's the case of unpleasantness. A product is so difficult and slow…

> This does not happen with products that are highly successful business-wise, no matter how bad the suffering, so it's really a business failure rather than a technical one.

One thing often feeds on the other. Because the system is hard to change, it does not get necessary features. Because it does not have necessary features, it provides less business value. Because it provides less business value, there is less of a budget for improving it. And so on.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#232
post #109

There are multiple cases of "killed by technical debt". There's the case of mysterious and unsolvable breakage. The product simply stops working, and the team is unable to get it working again, period. This can happen with really ancient legacy products where the original team is gone, or young products that are written badly by inadequate teams. There's the case of unpleasantness. A product is so difficult and slow…

I've learned over time that its always better to assume that who came before me were smarter than me and knew more than me (I'm rarely proved wrong).

Often it's the opposite.

They did something in an horrible way knowing it was horrible but because they have been asked to deliver a feature as soon as possible and at any cost.

This is a slippery slope. It lets a company move faster till it reaches the point that the software becomes an unmaintainable pile of hacks.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#233

Earlier quoted context omitted.

Yet another reason I swear by Heroku - you can rollback to the actual prior release, not rollback the code and try to rebuild.

Doesn't AWS do this too?

Which service? EC2 doesn't - at least not inherently, there might be a way to snapshot the machine's state; I'm not sure.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#234
post #109

There are multiple cases of "killed by technical debt". There's the case of mysterious and unsolvable breakage. The product simply stops working, and the team is unable to get it working again, period. This can happen with really ancient legacy products where the original team is gone, or young products that are written badly by inadequate teams. There's the case of unpleasantness. A product is so difficult and slow…

> This does not happen with products that are highly successful business-wise, no matter how bad the suffering, so it's really a business failure rather than a technical one. One thing often feeds on the other. Because the system is hard to change, it does not get necessary features. Because it does not have necessary features, it provides less business value. Because it provides less business value, there is less of…

Bingo.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#235
I had a chance to ship the first Tower Defense game for iOS. The OS X game I was porting had some crippling performance problems that were incredibly hard to track down.

The problem was two-fold:

1. The relevant tools (Unity3D) were extremely immature and the problem was quite diffuse. No profiler, poor quality of generated code, tiny caches, etc.

2. A problem in string-handling code that was quite diffuse throughout the game. As near as I can tell, it was blowing out the tiny CPU cache hundreds of times per frame.

On desktop, this code was a complete non-issue. On the puny little ARM on the iPhone? It was the difference between having dozens of towers and 50 enemies in play vs half a dozen towers and less than a dozen enemies in play. The impact on game dynamics, and need to re-balance everything by itself would add weeks to the shipping schedule.

There were plenty of other things that needed to be scaled WAY back of course: Switching from 3D to 2D to get vertex count and draw call count down. Completely rebuilding the entire UI. Revamping the pathfinding and suffix caching to not play havoc with the CPU cache. Moving from a 24x24 grid to a 12x12 grid. All of that combined helped a LOT, but not nearly enough.

The string manipulation was for a hierarchical property system that let me parameterize all sorts of attributes for enemies/spells/towers/projectiles in a set of text files. Ultimately, I had over-engineered on the assumption that I would be tweaking many more things -- with much greater frequency -- than I wound up actually tweaking.

Had I ripped most of it out and just had local properties on each prefab that I assigned manually, I might've hit that market opportunity. Finding that that was the cause was a multi-month project because of how interwoven it was with everything else. Hell, it would've been fine, had a not over-generalized it into a shared component on each prefab that the other components inquired with to get property values. But I did. And it took me vastly too long to identify it was the major problem it was.

Opportunity missed, and that was the final nail in the coffin for my fledgling game studio.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#236
post #188

Earlier quoted context omitted.

I've learned over time that its always better to assume that who came before me were smarter than me and knew more than me (I'm rarely proved wrong).

"They did it that way because they where stupid" is a ridiculously common assumption, when the correct answer is often "They did it that way because they knew stuff I don't know".

Well, the correct answer is often also "They did it that way because it was a reasonable choice then, and they didn't have the benefit of hindsight that I have now."

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#237
post #161

Earlier quoted context omitted.

Seen this a lot. A lot of companies think they are "product" companies, but due to their unwillingness to push back on customers, they become custom engineering shops, bolting on little one-off mods to their project over and over to appease bad customers (or to appease POTENTIAL customers who haven't even bought the product yet). Stop me when you recognize this one: "Hey your product is great, but we really want some…

I think every one of my former employers who have failed, did so by doing those 'customs.' The last one even spun off a dedicated team that built (hacked) prototype customs in order to secure sales, then threw away the prototype and, after collecting the commission, told the new customers that it would take several years to get what they just saw in production but in the meantime we can do our existing product with s…

Sounds like poor sales to me

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#238
post #128

Earlier quoted context omitted.

I should add here that the "antiquation" case is the one that has caused the most observed grief in my career. The forces causing failure are coming from outside the code/business (dependence on an outside vendor), and sometimes collide with forward momentum of other parts of the code (i.e. that graph library will never, ever work with Java 7, to name an example). These become life-or-death situations, and the tendri…

>You can still find teams dependent on an antique commercial version control system or IDE that greatly slows down or even stops work. Sounds like Rational ClearCase.

Yeah, that. But even things like cvs, that were modern and hip in 1998, are still floating around 20 years later.

I actually thought Subversion would be the last version control system, when it came out. Of course, now it's git. Maybe someday we'll get something better, and git will look decrepit.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#239
post #118

Earlier quoted context omitted.

I meant my follow-on to be a concession, but worded it poorly. I thought it had advantages, but yes, I was thinking small appliance mainly. In particular, in home. And not just computers, but lights and control panels. Seems many things all use the same power characteristics and are now becoming complicated by dealing with AC. Which, amusingly, is fitting for the tech debt debate. Eradicating some choices from the pr…

AC is much better in the home. There is no way to get around the fact that you need massive wires to supply low voltage at high amps. It is much cheaper to have a power supply on every electronic device turning 100-200 volts to 5 volts than to have one big power supply turning power line voltage to 5 volts. Of course a lot of computers need 3 volts or less, so the power supplies exist anyway. It is also more efficien…

>AC is much better in the home. There is no way to get around the fact that you need massive wires to supply low voltage at high amps.

That's orthogonal. What you really mean is that you want high(ish) voltage to distribute power in a home, in order to mimimize losses due to wire resistance over distances of dozens of meters.

You don't need AC to do that. In fact, with modern power electronics, the switching converters we now use for supplying LVDC to our devices can work just as well with DC as with AC input power.

The primary advantage of AC over DC is that it can be converted between voltage levels easily with transformers. But today, we can do the same thing with DC using DC-to-DC converters. These didn't really exist in an economical way before a couple decades ago, maybe even more recently.

If for some odd reason, western society decided to re-engineer and replace the whole power grid, it's quite likely I think they would simply switch to DC for everything. With deployment at that scale, the cost issues with the equipment should go away, making it no more expensive to replace everything with DC converters than transformers. DC is more efficient than AC because it stays at its peak voltage, and because it has no skin effect. But the technology needed to make it inexpensive to use for power transmission has only been around for a somewhat short time (namely, modern power electronics). Up until recently, it was simply a no-brainer to use AC because of its simplicity in generation, transmission (with transformers for stepping up the voltage), and usage (with AC motors).

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#240
post #238

Earlier quoted context omitted.

>You can still find teams dependent on an antique commercial version control system or IDE that greatly slows down or even stops work. Sounds like Rational ClearCase.

Yeah, that. But even things like cvs, that were modern and hip in 1998, are still floating around 20 years later. I actually thought Subversion would be the last version control system, when it came out. Of course, now it's git. Maybe someday we'll get something better, and git will look decrepit.

I have little doubt that git will be replaced eventually (or perhaps severely modified). It seems like a fad to me. It is indeed very powerful, but its UI is sheer insanity. At least SVN is very straightforward to use and understand. It just doesn't offer the distributed nature that git does, as it relies on a centralized server.
Post reply on HN