Ask HN: Have you ever worked on a product that was killed by technical debt?

I ran the project for the first four years. It was my first large project and the first time I had run a team of any size. I made a few mistakes that might be worth learning from, but my mistakes weren't the only ones responsible for the debt.

The largest driver in the technical debt issue was the timelines. My boss was new to software development, and his expectations were not in line with reality. We frequently had to ship ad hoc features under tight deadlines to keep him happy. When I pushed back on the timelines he became unhappy. So we acquired debt in the form of 1) hastily designed sections of code that don't lend themselves to scalability or refactoring 2) a lot of small code smell issues that by themselves don't amount to much but in the aggregate form a kind of surface scum that makes future development and more importantly testing difficult and thus brittle.

The debt, and the accompanying bugs and delays it caused, eventually led to enough dissatisfaction that I was taken off the project (I am in a weird outside position now, sort of half on half off, supporting a tangent of the software but not working on the main branch or involved in architecture discussions, planning, or execution of the new replacement). A new manager was hired. We disagreed over strategy, so I was taken off the project.

He wanted to start over from scratch, which is what they are essentially doing now. The plan is to pattern the new solution off the old one by incorporating all the business rules (which they want me to document), but they have very different implementations in mind. They don't want to reuse existing libraries (some of which is driven by a lack of familiarity or understanding of those libraries, how they work, why they approached the problem the way they did).

They face some significant challenges:

1) their coding velocity is slow. more than 60% of the original team has left and been replaced, so a large part of the team hasn't been on the project for more than 2-3 months. this means that there is a significant dearth of institutional knowledge. i think a decent pace is good for development of software, but they aren't starting from scratch, and there's a lot of expectations to meet.

2) because of the dearth of institutional knowledge when they do start implementation, they are going to end up repeating many of the mistakes made by the old team. I have limited insight into what those are, but based on what exposure i do have, i can see planned missteps all ready.

3) my boss, the owner of the company, has learned some patience, but they've been at the rewrite for nearly 6 months and have little to show for it. in terms of feature parity with the old software, they are severely lacking. i don't expect he is going to be willing to wait another full year to get an app that does essentially what he already has only differently, even if that architecture is more flexible.

In all honesty, I hope they succeed. My boss is a good friend and this experience hasn't ruined that. I have and am dealing with some resentments, but I don't want him to fail. And because this project was my baby (so to speak) there is a part of me that wants it to live.

----------------------

In retrospect, I'm not entirely sure what I should have done differently. Had I ignored the requests for adhoc features, I likely wouldn't have made it as far as i did, because without those features he would have pulled the plug. The company grew at an exponential rate the first 2 to 3 years of that project, and part of what fueled that growth was the adhoc, fast turn around times of my dev team. We were incurring debt, but we were also making big gains.

If I was going to do this all over again here is what I would do:

1) I would push back more, in smaller increments, placing more emphasis on getting the details right before shipping. 2) I would push back more on requirement creep. My boss would frequently have these "great" ideas that he would insist we work on, which would get half done then discarded leaving the code base littered with dead ends that needed cleaning up later. Part of the debt we acquired stemmed from the fact that in order to keep a lot of those activities from impacting ongoing efforts, i built an architecture that was somewhat disjointed. the lots of little islands approach meant we had apps that were reinventing the wheel, taking different approaches, etc... 3) Place a greater emphasis on automated testing (fewer human testors, more test engineers).

The other aspect to this is the fact that when you are creating software to solve problems that don't currently have software solutions, you spend a fair amount of time going down trails that don't pan out. The R&D aspect left us with bit and pieces of code in the code base that were incomplete or incompatible, but that were relied upon by one app or another.

We lost time discovering that certain approaches didn't work.

I think thats it... I hope someone finds the story useful.

As I understand it, "tech debt" is something that has to be reckoned with at some point, or else you're going to have real problems in the future (just like refusing to pay off a money debt will generally cause you real problems at some point when the creditor sues you and gets a judgment). You can't just let it go on forever; eventually you need to "pay it down" (by cleaning up the codebase, migrating to newer technologies, etc.), or else catastrophe happens (the company is unable to compete and goes under). One common factor cited in these stories is that the code becomes too unmaintainable and unreliable: too many weird changes for customers pile up and introduce serious bugs which cause the product to not work properly.

This isn't like that at all. We can go on with our current household AC power systems indefinitely. Maybe we could get a 1% improvement by switching to DC systems (at an enormous cost because most of your appliances and devices won't work with it without adapters), I don't really know exactly how much better DC would be (not much really), but what we have now works fine. Furthermore, it's not like the whole electric grid system needs to be changed: it's entirely possible, for instance, to switch distribution systems to DC and leave household systems AC. Instead of distributing the power at 30-something kVAC in your neighborhood and using outdoor transformers to step it down to 240VAC for your house, it could be distributed in DC form, and those transformers replaced by modules which convert the 30-something kVDC to 240VAC. In the old days, this was hard and expensive to do, but with modern power electronics it's not. But even here, the question is: are the gains worth the expense? And the answer is very likely "no". (For reference, I'm not a power engineer, I just studied it in college as a small part of my EE curriculum.)

So this does not, to me, resemble "tech debt" at all. It's just a system that we use for legacy reasons and which is extremely reliable and works well, even though it might not be the absolute most efficient way to solve the problem. This is no different than many other engineered systems. Perhaps you have a decent and extremely reliable car. Could it be better? Sure: you could build the chassis out of carbon fiber, use forged aluminum wheels instead of cast, etc. all to save weight and improve fuel economy. Are you going to do that? Of course not, because the cost is astronomical. There's cars like that now, and they cost $1M+.

So for AC systems that we're talking about, the question is: what is wrong with them that we want to consider replacing them with something else, instead of just sticking with them even if they're not quite as efficient as they could be? Because the cost to upgrade them would be enormous, so you need to have a very good reason.

Should I ever inherit an #ifdef mess again, I intend to replace #ifdefs with Strategy patterns.

#1 figure out all the known defs in actual use

#2 rerun the preprocessor with each variant (combo)

#3 capture the output(s)

#4 aggressively apply the Strategy pattern, refactor code

Last time, I removed dead code piecemeal manually. It sucked.

I suspect you could go a long way with the heuristic "If engineering asks for more process, always give it to them."

It's not flawless, but it's like hearing Ron Paul call for a new regulation - when a request is that out of character, you should usually suspect that there's some good motivation.