Live data from Hacker News

Ask HN: Have you ever worked on a product that was killed by technical debt?

news.ycombinator.com

321–330 of 331 posts

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#321

It seems everybody here is suggesting that they have worked on products that were killed by technical debt. I'm going to take a different stance and suggest it is very difficult for a product to be killed by technical debt. I'm working on a product now which has huge technical debt requiring a full re-write, but we have customers who love the product, so we just keep the old version going while building the new versi…

> You don't know that the new features would have saved it, Or could have killed it even faster. Imagine that at the beginning you decided not to accumulate "technical debt". Instead you got some product that developers believed had no "technical debt" and because of that it was late to the market. Money were running out and once customers started using it you had to be really lucky for new features to make enough im…

I think this point is very under appreciated, thanks for pointing it out.

Along with the 'feature' that could kill things faster is general bloat when people keep building without knowing what the customer wants. You have to try something, but product testing should be done in such a modular way that most things can be removed if it turns out to be the wrong direction. Of course, you need a base data structure, but features should conform to that rather than constantly extend.

One example I'm seeing with a start-up I currently know is that they are building SDKs for multiple languages. Most of the code is auto-generated, but just the time in examples, documentation and packaging is killing them while they don't have customers for most of the languages they're publishing. This isn't just 'code' debt, there is overhead in documentation and management of code.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#322

Yes and no. The project wasn't killed specifically because "you have technical debt". It was killed because there was no way for anyone to be effective with the combination of poor undocumented code. "We need to change the email message that goes out when someone registers". This took a team of (4?) people 5 calendar days to change. As a contractor, I had to vpn in to one system, then remote desktop over another vpn…

ACK - I FORGOT THE BEST BIT... (well, maybe not best, but...)

No one could install anything locally - everything had to be done on their locked down remote systems (some were Amazon remote desktops).

For the accessibility testing, the auditing company used JAWS. The company I was contracting to had one license (or so I was told) so I couldn't have one. We actually tried to install JAWS on an Amazon desktop, but it just crashed the entire virtual desktop, requiring re-imaging. That happened twice, so we gave up.

So, the proposed workflow was, I'd make a change, push code, email someone to move that code to a system that an internal tester could look at it. I'd get an email back, then email the internal tester that the code was ready to go look at. The internal tester would go to the screen(s) in question, using JAWS, then "tell me what JAWS said". That would often take several hours or a day.

I was then supposed to make changes based on that feedback, then repeat the cycle until things were 'fixed', then we'd ask the auditing company for another test, which they'd schedule for 2 weeks in the future. Then we'd wait.

During the first iteration of this part, sr mgrs kept asking me "when will this be done?". I kept trying to explain that we didn't even know what "done" was - the auditing company just had blind folks that would use the system with JAWS enabled and if they felt it was usable, they'd say so, otherwise, they'd report back "hey, this isn't usable", and we'd have to start digging in again.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#323
post #244

Earlier quoted context omitted.

IMO this is the price to pay for a dynamically typed language. 60K LOC is not much in a static language, you can use tools to refactor it easily or to visualize the control flow. But with a dynamically typed language? Its a nightmare. You change one thing and cannot possibly know what else could have gone wrong.

The legacy app has > 80% test coverage. Refactoring is still slow because there are all sorts of business assumptions put into place that add functionality without ever questioning the need for it. Basically, for a long time, the company never really re-evaluated what it had learned and spent time trimming things down, so as a result there is this ungodly mess. At the heart of what the business does, there is no real…

Unit tests just make the mess a bit easier to solve, but are far from a perfect solution. Wanna move and rename a function? Do it, then spend hours fixing your 10 broken tests, writing new ones, and test you app for hours because unit test dont cover integration. A static language + an IDE does it automatically within seconds.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#324
post #128
post #109

There are multiple cases of "killed by technical debt". There's the case of mysterious and unsolvable breakage. The product simply stops working, and the team is unable to get it working again, period. This can happen with really ancient legacy products where the original team is gone, or young products that are written badly by inadequate teams. There's the case of unpleasantness. A product is so difficult and slow…

I should add here that the "antiquation" case is the one that has caused the most observed grief in my career. The forces causing failure are coming from outside the code/business (dependence on an outside vendor), and sometimes collide with forward momentum of other parts of the code (i.e. that graph library will never, ever work with Java 7, to name an example). These become life-or-death situations, and the tendri…

> I've tech-led jumps to new version control systems a few times, and it's always riddled with anxiety, strain, and management angst. (And it always makes the team far happier and more productive!)

In the cases of this I've seen, it's always been because management and team priorities were unaligned.

Management in those places cared about a minimum level of productivity and minimizing risk.

Teams cared about maximizing productivity and their work days not sucking.

As long as teams kept managing to soldier through... rarely saw things change in those shops.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#325
I guess you could say this is a form of technical debt, or technical stubbornness.

I used to work for a company that specialized in Solar Panel data. I was hired to help the company catch up on their client queue (those waiting for the product to be set up).. and when I first started they were behind by about 140 clients and I had gotten the client queue down to about 30 clients remaining.

So our company would sell solar panels to corporations, and then they would put kiosks in the lobbys of their buildings so their clients could come take a look at the energy savings and other information about the building. It was a very popular way of doing things, especially in cities like New York, Chicago, and Los Angeles. Well, my job was actually designing the graphics for what was displayed on those kiosks and then syncing the data with the graphics.

There was a platform that did the work, but I still had to design the graphics and type in all the serial and model numbers and all that to sync everything together and make it show it in chart form in a gorgeous view. Anyways, for years, it was being done in Flash. Unfortunately, as progressive and dominant as our company was -- at one time, their technology was the best, even having received several awards. Over the years, HTML5, JSON, and other technologies were better and faster -- and could read data from a database just like as our system could, but turnaround time for production was faster for others. It took about 1 week for us to develop, set up, and sync everything from system to database. Now these other companies were able to get everything done in days, rather than a week or more. I think a huge part of the problem was my company had these corporations interested but forgot to hire enough people to actually do the work.

We did have an in-house software developer working on improving this technology, upgrading the software, but they seemed very reluctant to adapt the new software quickly. I had worked in the new software for a few days, and then all of a sudden, they called me into an office, and laid me off. A few months later, everyone else followed, and today, I think the company only holds on to about 2 or 3 employees who just maintain all the kiosks and software because of the existing contracts. Or maybe they went under... their website doesn't even work anymore.

Why did they go under? We were warned that technology was changing fast and that we needed to do something about it... and it was just the reluctance of the CEO to adapt the technology and push it out faster. Our competition slaughtered us. Luckily, it was my second job, so I worked out a great deal with them on letting me go.. I got about a month of vacation time on the condition that I wouldn't go file for unemployment. Obviously, already having another job, it worked out in my favor.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#326
post #314

Earlier quoted context omitted.

I wouldn't call it "tech debt". Present-day AC systems may not be completely optimal (given current electronics technology), but they do work well. As I understand it, "tech debt" is something that has to be reckoned with at some point, or else you're going to have real problems in the future (just like refusing to pay off a money debt will generally cause you real problems at some point when the creditor sues you an…

Most instances of tech debt are things you don't have to deal with. Usually, it is the term pulled out for things people don't like. Or generally deprecated methods that have better replacements, but still work. It is this second sense that I was latching on. It --tech debt-- will drive decisions today. But it is not clearly bad. Just a constraint on current decisions that was made in the past. Often for decent or re…

>Most instances of tech debt are things you don't have to deal with. Usually, it is the term pulled out for things people don't like. Or generally deprecated methods that have better replacements, but still work.

I'm not so sure about this. To me, "debt" is something that has to be paid eventually. Otherwise, why use the term "debt" at all?

So if something works fine, why waste your time and energy replacing it with something newer?

Usually, the reason for this is the assumption that sticking with something deprecated will eventually bite you in the ass: something you're depending on won't be supported, will have security holes that won't get fixed, etc., and you're going to wish you had fixed it earlier. So this is a valid use of the term "tech debt" IMO.

But if something is just something someone doesn't like, that isn't "tech debt" at all. I don't like .NET, but it's invalid for me to call all software written in .NET "tech debt". I don't like Apple's ecosystem, but it would be pretty ridiculous for me to call all iOS software and apps "tech debt" when many millions of people use and enjoy that software every day.

So, for your LaTeX example, I don't consider that tech debt at all; instead, it's just like iOS and .NET software to me. If someone doesn't like it, that's their problem; the fact that it isn't brand new isn't a problem for me and all the people who still happily use it.

So personally, I think anyone using the term "tech debt" to just refer to things they don't like is using it incorrectly and in a totally invalid way.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#327

Earlier quoted context omitted.

AC is much better in the home. There is no way to get around the fact that you need massive wires to supply low voltage at high amps. It is much cheaper to have a power supply on every electronic device turning 100-200 volts to 5 volts than to have one big power supply turning power line voltage to 5 volts. Of course a lot of computers need 3 volts or less, so the power supplies exist anyway. It is also more efficien…

>AC is much better in the home. There is no way to get around the fact that you need massive wires to supply low voltage at high amps. That's orthogonal. What you really mean is that you want high(ish) voltage to distribute power in a home, in order to mimimize losses due to wire resistance over distances of dozens of meters. You don't need AC to do that. In fact, with modern power electronics, the switching converte…

> it's quite likely I think they would simply switch to DC for everything

I'm not sure. AC has some important safety considerations that would make it better even if the efficient was significantly worse.

Switches, fuses and circuit breakers that work with DC are more expensive than AC. When a circuit opens there is a spark, and this spark can in some cases create a conductive plasma. With AC the wave goes to zero and the plasma disappears, while with DC it continues. There are cases where a DC fuse blew but the fuse continued to conduct. Of course this can be engineered around, but generally with larger and more expensive parts.

When someone touches power accidentally, AC is slightly safer. With DC your muscles will grab and never let go. AC gives you a chance to let go. This is a low probability thing, but is a factor.

The guy who wanted us to debate is wrong for one other reason though: I'm approaching the limits of what I know on the subject, while you seem to have a lot more knowledge.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#328
post #314

Earlier quoted context omitted.

Most instances of tech debt are things you don't have to deal with. Usually, it is the term pulled out for things people don't like. Or generally deprecated methods that have better replacements, but still work. It is this second sense that I was latching on. It --tech debt-- will drive decisions today. But it is not clearly bad. Just a constraint on current decisions that was made in the past. Often for decent or re…

>Most instances of tech debt are things you don't have to deal with. Usually, it is the term pulled out for things people don't like. Or generally deprecated methods that have better replacements, but still work. I'm not so sure about this. To me, "debt" is something that has to be paid eventually. Otherwise, why use the term "debt" at all? So if something works fine, why waste your time and energy replacing it with…

I find this a compelling view. But, I urge you, just google technical debt. You will see the definition: "Technical debt is a concept in programming that reflects the extra development work that arises when code that is easy to implement in the short run is used instead of applying the best overall solution."

So, in this case, AC/DC fits if we agree there is a chance the "best overall" solution is DC. (Which, I fully grant, is not a given.) There is also a bit of playing loose with "short run."

Then, skip back to the top of this thread, where you will find: "products that are written badly by inadequate teams" and "case of unpleasantness" and "A product is replaced (or intended to be replaced) by a new product that does more or less the same thing, only this time with a smart new team, in a hip new language..."

All of this is the first, most highly voted, post. The next post is a highlight of poorly engineered solutions.

My point? Find a case study that has the usage you are referring to here.

Now, certainly rhetorically it has this appeal to people. But I have never seen it used in a way that it fits the metaphor. Just used to hit the emotional strings of "you must pay back your debt!" While usually claiming that the design or lack of some technology is the debt.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#329
post #328

Earlier quoted context omitted.

>Most instances of tech debt are things you don't have to deal with. Usually, it is the term pulled out for things people don't like. Or generally deprecated methods that have better replacements, but still work. I'm not so sure about this. To me, "debt" is something that has to be paid eventually. Otherwise, why use the term "debt" at all? So if something works fine, why waste your time and energy replacing it with…

I find this a compelling view. But, I urge you, just google technical debt. You will see the definition: "Technical debt is a concept in programming that reflects the extra development work that arises when code that is easy to implement in the short run is used instead of applying the best overall solution." So, in this case, AC/DC fits if we agree there is a chance the "best overall" solution is DC. (Which, I fully…

I think we're going off on a tangent here, but even with that definition from Wikipedia, there's no such thing as "the best overall solution". Everyone is going to disagree about that; the best you'll get is a consensus. For instance, back to LaTeX, there's countless academics out there who use TeX/LaTeX/whateverTeX for writing academic papers, and getting beautiful results while not having to mess around with a WYSIWYG editor like MS Word and just typing in some simple formatting codes. That's what *TeX was designed for and has worked well for for ages. But I'm sure you'll find a few people who say this is bad because it's "old" and that they should switch to the latest MS Word for everything, and rewrite all their papers in the latest MS Word. If you look really hard, you might even find someone who thinks both are bad, and that all academics should rewrite everything in WordStar.

"The best overall solution" is up for debate. It's the same with programming languages; one team will say that C is the best overall solution for a certain problem, another team will say it's Python, another team will say it's one of the .NET languages. I'm sure you can find plenty of engineers who will claim that mission-critical real-time avionics systems or automotive ABS controllers should be redesigned to use x86 CPUs and run Windows and have the code written in C# instead of using C/C++ and running on a small RTOS on an embedded microcontroller.

The implication I see with your Wikipedia definition is that implementing something easy in the short run instead of something that really is the best overall solution will eventually lead to more work to fix the shortcomings of the quick-n-easy solution. So, like I said before, a "debt", because it has to be paid back eventually (with work). The problem I see is that not everyone agrees on what is the best overall solution, and unlike a money debt that's easily seen by looking at a dollar figure, the only way to really know how much "tech debt" you have is through experience, i.e. accumulating it and then finding out over time how much work you have to expend to fix things when your quick-n-easy solutions start having real, demonstrable problems. If your solution has no actual, demonstrable problem (e.g., you use LaTeX and it continues working great year after year for your use-case), then I don't consider that to be "tech debt" at all, even if some people don't like it.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#330
post #328

Earlier quoted context omitted.

I find this a compelling view. But, I urge you, just google technical debt. You will see the definition: "Technical debt is a concept in programming that reflects the extra development work that arises when code that is easy to implement in the short run is used instead of applying the best overall solution." So, in this case, AC/DC fits if we agree there is a chance the "best overall" solution is DC. (Which, I fully…

I think we're going off on a tangent here, but even with that definition from Wikipedia, there's no such thing as "the best overall solution". Everyone is going to disagree about that; the best you'll get is a consensus. For instance, back to LaTeX, there's countless academics out there who use TeX/LaTeX/whateverTeX for writing academic papers, and getting beautiful results while not having to mess around with a WYSI…

I 100℅ agree regarding "best overall solution.". Indeed, in large that is my point.

Alternatives may have advantages. However, often the advantages of where one is at are ignored in the debate.

My gripe in this debate is more from actual uses of the term. Not from any ideal use of it.

Post reply on HN