Live data from Hacker News

Ask HN: Have you ever worked on a product that was killed by technical debt?

news.ycombinator.com

301–310 of 331 posts

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#301
post #292

Earlier quoted context omitted.

I'd be interested in you two debating this more, since you both clearly know the topic better than I do. This post is reflecting what I thought I had heard. But, I am not in this field.

There's really nothing to debate; the guy I replied to was totally correct about everything except the bit about "AC is much better in the home", where I pointed out that he really meant that a high voltage roughly where our current AC systems are (120V-240V) is much better in the home than some kind of low-voltage DC system, and that with modern technology, it would probably actually be better to have a DC system. B…

I was never arguing that an individual should replace the AC in their house. My argument was, with current technology, the AC setup can be seen as tech debt.

Which seems compatible with what you are saying, but the parent was specifically claiming I was wrong.

That is, you seem to be echoing my point. But seem to be claiming it is different. What am I missing?

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#302
I recently left a company with a software product that may die due to a misdiagnoses of tech debt. They are in the midst of an unnecessary rewrite with nothing to really show for it after almost a year.

The previous project has some issues but they were fixable with refactoring while continuing feature dev. The rewrite was done over my objections. New management had been hired and they were looking to make their mark. I left before it could all come crashing down.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#303

Yes and no. The project wasn't killed specifically because "you have technical debt". It was killed because there was no way for anyone to be effective with the combination of poor undocumented code. "We need to change the email message that goes out when someone registers". This took a team of (4?) people 5 calendar days to change. As a contractor, I had to vpn in to one system, then remote desktop over another vpn…

I think this is classed as lethal technical debt

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#304

Yes and no. The project wasn't killed specifically because "you have technical debt". It was killed because there was no way for anyone to be effective with the combination of poor undocumented code. "We need to change the email message that goes out when someone registers". This took a team of (4?) people 5 calendar days to change. As a contractor, I had to vpn in to one system, then remote desktop over another vpn…

This is a great example of how technical debt 'kills'. It's not a murder, it's negligence and a slow demise. I went through one of these projects. The tech debt was never as bad as you describe, but it was a small company operating on a short runway. It also taught me an unfortunate lesson about non-technical founders and the dangers of outsourced code. The MVP for the company had been bought off the shelf. It worked…

This part in Rich Hickey's Simple Made Easy talk [1] had a lasting impression on me. It really drove home the point on how a build up of complexity (one of the most common forms of tech debt, and one of the hardest to avoid) can eventually "kill" a project in exactly the way you described, slowly and painfully:

    "But I have all this speed. I'm agile. I'm fast. You know, this easy stuff is making my life good because I have a lot of speed."

    What kind of runner can run as fast as they possibly can from the very start of a race?

    [Audience reply: Sprinter]

    Right, only somebody who runs really short races, okay?

    But of course, we are programmers, and we are smarter than runners, apparently, because we know how to fix that problem, right? 
    
    We just fire the starting pistol every hundred yards and call it a new sprint.

    ...It's my contention, based on experience, that if you ignore complexity, you will slow down. 
    
    You will invariably slow down over the long haul.

    ...if you focus on ease, you will be able to go as fast as possible from the beginning of the race. 

    But no matter what technology you use, or sprints or firing pistols, or whatever, the complexity will eventually kill you. 

    It will kill you in a way that will make every sprint accomplish less. 
    
    Most sprints will be about completely redoing things you've already done. 
    
    And the net effect is you're not moving forward in any significant way.
[1] https://github.com/matthiasn/talk-transcripts/blob/master/Hi...

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#305
post #199

I was CTO of a company that had a two-week outage due to technical debt. I didn't sleep much for any of it. We fixed it, and we'd lost about 30% of our subscriber base in that period. The company took on new funding to survive, invested that in a new set of products, and shuttered the old stuff just to stay afloat. I am currently working in a business where there is a nearly 8-year old Rails app (600+ models, 250+ co…

Can you expand on how technical debt caused a two-week outage?

There are two problems with giving a full answer: firstly I'm still under NDA for some of that, and too much detail would breach that; secondly, my exact memory of it is limited.

In short, my predecessor had attempted a move to SOA without understanding dependencies, circuit breaking and failure modes. This would then cause scenarios where the entire front-end would fail to render on a single down-stream service taking a little longer than necessary.

When identifying how to stop that happening, I discovered a large number of comments tagged "TODO" with statements like "Refactor this when we have time" or "We need to find a way to do this better".

Further down on the downstream services there were rather esoteric SQL queries doing large joins that nobody had done a query plan on. It was hard to identify these because the ORM had been trusted to do magic, and it was happy to do so, but there was a point where it was not apparent _why_ these joins were happening, but when you found the code, there were more comments "This needs improving", "We should refactor this", etc.

We were able to get something back quite quickly with liberal application of indexes, and it took us a day or two to refactor the queries enough to mean response times came down, but the error rate was still > 20%, and it was random, so 1-in-5 page loads of the front end service would fail.

We refactored the code to circuit break and handle degraded services better, but that took a few days, and then we started working down to the back end service and figuring out the final steps.

It was a small team looking after legacy code that everybody knew was a bit messy.

A few weeks before this code was shuttered, I heard from a friend that some of our content did not render at all on certain Android devices. I identified the cause as a half-finished refactor (again, my predecessor), that had never been finished because he had been pushed to work on something else. This caused a dramatic decline within a key market segment that resulted in declining ad revenue, subscriptions and overall viability of the business.

Basically, when you start something, finish it. If you find yourself putting in comments like "We should refactor this" anywhere in your code base, and you're doing so because the business is pushing you to work on new features, you have a massive problem culturally that is going to cause a rise in technical debt that raises risk to revenue.

All technical debt ultimately will lead to problems that the business will see on balance sheets, but they will rarely successfully identify the cause as being technical debt because they can't see, understand or rationalise it. They think it's engineers being grumpy idealists.

People play too fast and loose with the concept of "MVP" for my tastes, and it's a problem I see over and over again. The risk of that is, long-term, it will cause business failure.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#306
post #244

I was CTO of a company that had a two-week outage due to technical debt. I didn't sleep much for any of it. We fixed it, and we'd lost about 30% of our subscriber base in that period. The company took on new funding to survive, invested that in a new set of products, and shuttered the old stuff just to stay afloat. I am currently working in a business where there is a nearly 8-year old Rails app (600+ models, 250+ co…

IMO this is the price to pay for a dynamically typed language. 60K LOC is not much in a static language, you can use tools to refactor it easily or to visualize the control flow. But with a dynamically typed language? Its a nightmare. You change one thing and cannot possibly know what else could have gone wrong.

The legacy app has > 80% test coverage. Refactoring is still slow because there are all sorts of business assumptions put into place that add functionality without ever questioning the need for it.

Basically, for a long time, the company never really re-evaluated what it had learned and spent time trimming things down, so as a result there is this ungodly mess. At the heart of what the business does, there is no real need for more than a dozen models. So why do we have so many more? Nobody ever refactored away stuff we didn't need any more, and so weird things happen.

There is also a coupling issue that is endemic to all monoliths. We're moving to a micro-service architecture with clean domain separation, and we'll probably go to 1/10th of the code base in LOC terms within 12 months, even if we move some of that functionality into Go, Java or Python services (all options).

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#307

I was CTO of a company that had a two-week outage due to technical debt. I didn't sleep much for any of it. We fixed it, and we'd lost about 30% of our subscriber base in that period. The company took on new funding to survive, invested that in a new set of products, and shuttered the old stuff just to stay afloat. I am currently working in a business where there is a nearly 8-year old Rails app (600+ models, 250+ co…

I would say that 8 years to write 60k LOC is slow. I worked as the sole GUI software engineer for a hardware firm and wrote > 100K LOC in 3 years, not including the test projects proceeding the actual real project. This was in C++, and included client/server stuff, entirely custom resizeable GUI, OpenGL 3D graphics and modelling of 3D assets and textures etc too. And getting it running under OSX + Win32, fixing issue…

This is why LOC is a terrible metric. I was using it as a barometer as most ruby devs would see 60k LOC and go "Ooooookaaaay...."

If anything, we've gone too fast and not spent enough time going back and understanding what we really need to keep.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#308

Earlier quoted context omitted.

There's obviously a lot of stuff I don't know then. Like the benefits of copy pasted code, or 300 column lines, or implementing the logic in 20 places when it has existed in the standard library for a decade. If only the ancient sage I inherited this code base from had left notes to guide me on this path of wisdom.

Copy-pasted code: Data redundancy 300 column lines: Support for management buying everyone nice, new, giant monitors. Logic in 20 places: But what if we want subtle differences between each implementation? On a more serious note, a product that I've worked on was started in about 1998 in C++. We support something like 15 different platforms, and we've got our own implementations of things like vectors because we need…

Libreoffice managed to get rid of their legacy containers and moved to the STL, so maybe you can do it too

https://people.gnome.org/~michael/data/2011-05-12-libre.odp

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#309
post #199

Earlier quoted context omitted.

Can you expand on how technical debt caused a two-week outage?

There are two problems with giving a full answer: firstly I'm still under NDA for some of that, and too much detail would breach that; secondly, my exact memory of it is limited. In short, my predecessor had attempted a move to SOA without understanding dependencies, circuit breaking and failure modes. This would then cause scenarios where the entire front-end would fail to render on a single down-stream service taki…

Solid lesson. Thank you for sharing.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#310

Earlier quoted context omitted.

I have little doubt that git will be replaced eventually (or perhaps severely modified). It seems like a fad to me. It is indeed very powerful, but its UI is sheer insanity. At least SVN is very straightforward to use and understand. It just doesn't offer the distributed nature that git does, as it relies on a centralized server.

The cool thing about Git is that the underpinnings ultimately boil down to a key-value data store. I'm not an expert on these innerworkings, but in theory there's nothing stopping someone from creating a new UI that maintains most or all of the same strengths, except that everyone already uses and is used to the current way. I suspect if you came out with "SuperVCS" that was ultimately just a new UI on Git you'd have…

Isn't that basically Gitless? (http://gitless.com/)
Post reply on HN