Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Have you ever worked on a product that was killed by technical debt?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 331 posts

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#81

Post by luca_ing »

I'm currently working on the reincarnation of a project that was killed by technical debt -- TWICE.

The original codebase was about 20 years old. It was control code for something best described as an industrial robot. Written for the last 20 years by greybeards who knew a lot about the manufacturing process, and were reasonably good at getting a product out the door.

But the whole thing was riddled with #ifdefs for this customer or that, or one batch of machines or another. All long forgotten, written by people who had since left, or been pensioned. It was in dire need of improvement and extension, but it would have been superhuman to inject new features into this rat's nest. Plus their electronics supplier was discontinuing the control electronics the system was designed for. The UI also looked like it had been designed by German engineers in the 1980s. Which was the case.

So they made the defensible decision to start from scratch. A team of engineers was to develop an brand new machine, with all new electronics and all new code. They got to work -- and had to scrap the new software about three years in. It was just utterly misdesigned, and riddled with bugs. It featured wonderful WTFs like the embedded realtime code depending on the Qt libraries.

I observed its instability myself: it would just spontaneously crash every five minutes, sometimes just while idling. Once the project lead was on holiday, the programmers revolted, went to the head of the company, and the project lead found himself without a project on his return. Whee.

Now we've started from scratch again, and have at least succeeded in making different mistakes this time around. Fingers crossed, this might end up working.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#82

Post by hyperpape »

ekidd wrote: post #60

Projects rarely die because of technical debt. Instead, it becomes ridiculously expensive and difficult to add new features. But the software itself can remain in use for decades, gradually decaying and rarely adapting to changes in the business environment. Eventually either the software gets thrown out and replaced with something new, or the company is no longer able to compete. I've seen this play out probably clo…

This is only true if developing new features isn't part of how the company succeeds. That's probably true for some tools that are used internally. If you can't modernize your payroll, that might cost you some money, but it's not make or break.

For a company that makes software as a product, or to directly support or create their main product, not being able to add new features is a really bad place to be.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#83

Post by VLM »

I've been on one lack of loose coupling project. "All you're doing is importing from here and exporting to there, no problem, right?" and eighteen months later its somehow expanded into touching every part of the business except the vending machine firmware (no, was not a vending machine company). This leads to paralysis where touching anything makes anything and everything randomly crash. This was fundamentally a business process failure in that you can trivially specify a process too big for humans to implement, or even tool assisted humans. However the failure mode of technical debt could have hit a little less suddenly and catastrophically via some technical design decisions (should have jumped off the rails at 6 months and been redirected instead of patching around with the result of exploding at 18 months, etc).

I've been on two lack of refactoring/standardization projects. If you use rails, like back in the ver 1.0 era, you can't just stop updating and do something else, you have a tiger by the tail and if you don't keep up you'll never, ever, be able to catch up ever again. You can't go from 1.0 era to today where today is any time in the last five years. Scrap and complete rewrite. Of course management doesn't understand dependency trees and going back to OS and libraries from 2007 means rolling back 10 years of security patches or hand compiling everything and it would be a lot simpler to just rewrite.

I've been tangentially involved in a poor leadership situation where basically an entire department was forced out by a new leader, taking all their domain specific knowledge with them, and then the consultant friends brought in bled the company dry killing it in a race between expenses of consultants and smelly code being unusable. On paper the company died because of the financial load of switching completely over to outsourcing ("We're not a software company so we will not have developer employees anymore ... but we will have twice as many consultants working two thousand hours per year for five times the pay temporarily")

I've never experienced the parallel development trap, or lack of test suite, those must be interesting.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#84

Post by corecoder »

pards wrote: post #33

Knight Capital lost $465 million in 45 minutes caused (at least in part) by technical debt and poor development practices. Summary: http://pythonsweetness.tumblr.com/post/64740079543/how-to-lo...

To me, the single line that stood out was:

"The new RLP code also repurposed a flag".

I've never seen a flag repurposed without catastrophic effects.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#85

Post by coreymaass »

Yes. As a solopreneur, I had a moderately successful side project. When the app fell over, it took me two weeks to get it back online due to compounded tech debt. As a daily service, I lost most of my users when the app didn't work for 14 days. The app never recovered, and ultimately I sold it. One more vote against doing everything yourself.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#86

Post by logfromblammo »

If the company has a government contract, the technical debt becomes a hidden feature.

Government customers rarely have the ability to independently assess what something should cost if done according to industry best practices. So what you do is bid low, with an extremely short time horizon to release. You get the contract, and now have license to run up a lot of technical debt, because it needs to be done fast and cheap.

Now you get to maintenance phase. That's where the money is. The typical government customer is never willing to spend even one penny on paying off the principal on technical debt, but will make the installment payments forever. They will spend $100k on one new feature, but not even $0.01 on reducing the price of new features. Many still use SLOC as a management metric. So you run your codebase up to a million lines of code, when the software itself is just another glorified CRUD app. For bonus points, you give yourself 100% test coverage on a bunch of functions that have boolean "isUnitTest" parameters.

I imagine this is similar to how VC firms loot companies by manipulating their financial structure. I find it to be extremely unethical, but there is literally nothing I can do individually to put a stop to it.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#88

Post by CydeWeys »

taeric wrote: post #80
misja111 wrote: post #13

It happened to me twice. The first time was in a start-up at the beginning of the century, we were developing an electronic health record and we had outsourced the database abstraction layer to a company in Greece. In the beginning things went fine but after a while the development of the DAL went slower and slower and it became unstable as well. Eventually the word came out: the main developer of the DAL framework h…

This sounds less like technical debt, and more like liabilities of over engineering. Possibly feature creep. That is, technical debt is not necessarily tangled over-engineered code. It is more compromises that were made to actually ship and operate in the world. You can see this in the world with devices. Consider, technical debt is the reason you have AC delivered to your house going through as many converters as yo…

> Consider, technical debt is the reason you have AC delivered to your house going through as many converters as you do devices. Often to the same target power characteristics for those devices.

Bad example. AC power has many desirable characteristics for the local transmission grid. If you were to do the grid over from scratch you'd still use AC. You're also too focused on household electronic usage, which is a very tiny percentage of the overall electricity used.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#89

Post by taeric »

CydeWeys wrote: post #88
taeric wrote: post #80

Earlier quoted context omitted.

This sounds less like technical debt, and more like liabilities of over engineering. Possibly feature creep. That is, technical debt is not necessarily tangled over-engineered code. It is more compromises that were made to actually ship and operate in the world. You can see this in the world with devices. Consider, technical debt is the reason you have AC delivered to your house going through as many converters as yo…

> Consider, technical debt is the reason you have AC delivered to your house going through as many converters as you do devices. Often to the same target power characteristics for those devices. Bad example. AC power has many desirable characteristics for the local transmission grid. If you were to do the grid over from scratch you'd still use AC. You're also too focused on household electronic usage, which is a very…

My understanding is that HVDC had advantages. That said, I was also intending that to include the distribution in your house.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#90

Post by samwilliams »

byroot wrote: post #77
tboyd47 wrote: post #66

The closest I've come was a Rails project I inherited from a star developer who had just left the company. It was a B2B project that involved importing large Excel spreadsheets of various different formats into a standardized database for itemized review. The code was pretty sloppy, but didn't deviate much from standard Rails idioms. Not many people on the team understood Rails well enough to read it, but I did. Bug…

I don't think I ever seen an Excel / CSV import implementation that wasn't a huge mess.

Do you mean Excel to CSV? Or a CSV importer?

I totally agree about Excel importing, but CSV is trivial, no? Here is an Erlang version I happened to write yesterday:

  lists:map(
    fun(Row) -> string:tokens(Row, [SepChar]) end,
    string:tokens(InputStr, "\n")
  ).
EDIT: I know this version won't support escaped separator/newline characters, but I made it for a specific use case in which I knew that would not occur. Adding that functionality would make it a little messier, but still not too bad.

EDIT2: Thanks for the interesting comments! Not so trivial after all!

Perhaps a more accurate version of what I was attempting to say above is that 'it is often (not always) easy to build a CSV parser to interact with one specific program'. The four line version above works perfectly for reading the type of files I designed it for. If you want to work with human created, or more complex variants of CSV, all bets are off.

Post reply on HN