Ask HN: Have you ever worked on a product that was killed by technical debt? • Hacker News Board

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#251

Post by andrewclunn » Tue, Jan 17, 2017, 7:36 PM UTC

ryandrake wrote: post #107

Bartweiss wrote: post #93

Earlier quoted context omitted.

This is a great example of how technical debt 'kills'. It's not a murder, it's negligence and a slow demise. I went through one of these projects. The tech debt was never as bad as you describe, but it was a small company operating on a short runway. It also taught me an unfortunate lesson about non-technical founders and the dangers of outsourced code. The MVP for the company had been bought off the shelf. It worked…

Seen this a lot. A lot of companies think they are "product" companies, but due to their unwillingness to push back on customers, they become custom engineering shops, bolting on little one-off mods to their project over and over to appease bad customers (or to appease POTENTIAL customers who haven't even bought the product yet). Stop me when you recognize this one: "Hey your product is great, but we really want some…

Working for a company like that, BUT they allowed me to completely rewrite 3 of the tools from scratch in a more modular fashion so that I could do these things without having to modify the old code bases. Now there are two other applications that I still have to support (and were written by a consulting company we no longer contract through). It's night and day. So this isn't really the worst thing if you're given the authority and power to take full control of an application and rebuild it and take ownership of it. Of course, this doesn't really apply to junior devs.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#252

Post by wheaties » Tue, Jan 17, 2017, 7:40 PM UTC

72deluxe wrote: post #201

PaulRobinson wrote: post #191

I was CTO of a company that had a two-week outage due to technical debt. I didn't sleep much for any of it. We fixed it, and we'd lost about 30% of our subscriber base in that period. The company took on new funding to survive, invested that in a new set of products, and shuttered the old stuff just to stay afloat. I am currently working in a business where there is a nearly 8-year old Rails app (600+ models, 250+ co…

I would say that 8 years to write 60k LOC is slow. I worked as the sole GUI software engineer for a hardware firm and wrote > 100K LOC in 3 years, not including the test projects proceeding the actual real project. This was in C++, and included client/server stuff, entirely custom resizeable GUI, OpenGL 3D graphics and modelling of 3D assets and textures etc too. And getting it running under OSX + Win32, fixing issue…

As a former C++ dev, you can't compare Ruby LOC to C++ LOC. Just 4x the Ruby and then it becomes more fair. C++ is just verbose. #import doesn't fix everything blocks fix.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#253

Post by michaelfeathers » Tue, Jan 17, 2017, 7:42 PM UTC

Products are more often sidelined by technical debt than they are killed by it.

A typical scenario is:

1) Product becomes too hard to change but has many existing customers

2) Product is off-shored, and costs of maintenance are under-estimated because it depends on a vast technical ecosystem

3) Alternative products are developed to replace revenue of the original

The original product can live in this moribund state for decades.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#254

Post by kernelbandwidth » Tue, Jan 17, 2017, 7:43 PM UTC

jrs235 wrote: post #175

kernelbandwidth wrote: post #149

Earlier quoted context omitted.

We're an enterprise software shop, which necessarily means we do a lot of custom work, but we're careful to consider what we'll do. My mentor is an old hand that been through multiple exits, and in every meeting we have, he hammers this point. You're either a product shop, or a professional services shop, and if you don't know which one you are (or you believe wrongly) you die. Simple as that. The deeper you get into…

Companies that think they are a product shop but chase enterprise customers and do professional services often fall to appropriately charge enough for their services. Enterprise level customers require not only more features, more guarantees, and more support, then require more attention. Are you appropriately including sales time and expense chasing them to get a contract as well as support resources into your CAC?…

Yes, this. One of the big things I talk about with sales is the difference between changing a priority for a customer vs. adding distinct new things. As a recent example, a client wants better and faster feedback on the trial they're conducting (we're in the med-tech space), and we've already got a new dashboard designed and on our product roadmap. I'm more than happy to prioritize that over other product pieces if it'll get us the contract, because we're already going to do it, we're only changing the 'when'.

On the other hand, when they ask for something off the roadmap, we get into more complex issues (is this market-demand data, or custom work?) Particularly for grunt-level custom work (say, adding a support for tracking data on a niche wearable device that we don't currently support) there's a lot more questions that follow.

One of the most insidious of the latter, IMO, is that if it's just for one contract, then we're either hiring contractors/outsources (expensive, high management overhead), hiring new engineers (risky to grow headcount on a whim), or redirecting resources to tasks that are likely to have both lower ROI and provide lower growth for the re-tasked engineer. At our small size and need for high-quality people, I consider this to be a real cost too.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#255

Post by mason55 » Tue, Jan 17, 2017, 7:46 PM UTC

krmboya wrote: post #224

beat wrote: post #109

There are multiple cases of "killed by technical debt". There's the case of mysterious and unsolvable breakage. The product simply stops working, and the team is unable to get it working again, period. This can happen with really ancient legacy products where the original team is gone, or young products that are written badly by inadequate teams. There's the case of unpleasantness. A product is so difficult and slow…

Would these be cases where Robert Martin's 'The Clean Architecture' [1] would help, where the core enterprise logic is separated from third party dependencies, making the latter easy to swap out and replace? I'd imagine a number of these cases are caused by a heavy reliance on third party technologies that are no longer supported, or very few people still understand. [1] https://8thlight.com/blog/uncle-bob/2012/08/13…

It's a great idea but in reality most third-party tools are going to work slightly differently and the abstractions will leak (unless they were developed against an existing interface, in which case you don't need to create the wrappers yourself anyway).

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#256

Post by bloaf » Tue, Jan 17, 2017, 7:48 PM UTC

lutorm wrote: post #236

beat wrote: post #188

Earlier quoted context omitted.

"They did it that way because they where stupid" is a ridiculously common assumption, when the correct answer is often "They did it that way because they knew stuff I don't know".

Well, the correct answer is often also "They did it that way because it was a reasonable choice then, and they didn't have the benefit of hindsight that I have now."

Its also possible that whatever constraint they were working around no longer exists. Either way, it's a case of not tearing down a fence before you know why it was put up.

G.K. Chesterton, 1929:

>In the matter of reforming things, as distinct from deforming them, there is one plain and simple principle; a principle which will probably be called a paradox. There exists in such a case a certain institution or law; let us say, for the sake of simplicity, a fence or gate erected across a road. The more modern type of reformer goes gaily up to it and says, “I don’t see the use of this; let us clear it away.” To which the more intelligent type of reformer will do well to answer: “If you don’t see the use of it, I certainly won’t let you clear it away. Go away and think. Then, when you can come back and tell me that you do see the use of it, I may allow you to destroy it.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#257

Post by novaleaf » Tue, Jan 17, 2017, 7:49 PM UTC

I owned the project and decided to kill it, I guess you can call it due to technical debt, but probably more accurately it was due to incompetence, both by the developers and by myself as their "manager".

my product was http://www.teamkpi.com/

I hired 3 mid-level PHP and jsp developers in Thailand and had them make the website + reporting page.

total nightmare. Don't hire developers and assume that they will rise to the occasion (learn new tricks). I gave them as much time as they needed to research and make sound engineering decisions, I ended up with a spaghetti nightmare Frankenstein mix of server side scripts mixed with client side script mixed with server side that generates client side script.

In Thailand at least, you always need a manager to force architecture and design decisions, and force devs to refactor poorly thought out solutions.

I was naive and thought that I could have a team of 3 figure out the web part while I write the desktop client and provide PM-level guidance.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#258

Post by cthulhubot » Tue, Jan 17, 2017, 7:50 PM UTC

jrs235 wrote: post #175

kernelbandwidth wrote: post #149

Earlier quoted context omitted.

We're an enterprise software shop, which necessarily means we do a lot of custom work, but we're careful to consider what we'll do. My mentor is an old hand that been through multiple exits, and in every meeting we have, he hammers this point. You're either a product shop, or a professional services shop, and if you don't know which one you are (or you believe wrongly) you die. Simple as that. The deeper you get into…

Companies that think they are a product shop but chase enterprise customers and do professional services often fall to appropriately charge enough for their services. Enterprise level customers require not only more features, more guarantees, and more support, then require more attention. Are you appropriately including sales time and expense chasing them to get a contract as well as support resources into your CAC?…

This is a frighteningly accurate description of the company I'm currently at. They spent many years chasing after the enterprise level customers at the cost of alienating their smaller team level users and never had an answer when requests would crop up from the larger accounts asking for features ('just get it done'). Now they're trying to pivot back to the team level customers and are having an supremely difficult time dealing with the tech debt built up by addressing the enterprise level concerns. We tout ourselves as being a product shop when in reality we're trying to be both.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#259

Post by jrs235 » Tue, Jan 17, 2017, 7:51 PM UTC

kernelbandwidth wrote: post #254

jrs235 wrote: post #175

Earlier quoted context omitted.

Companies that think they are a product shop but chase enterprise customers and do professional services often fall to appropriately charge enough for their services. Enterprise level customers require not only more features, more guarantees, and more support, then require more attention. Are you appropriately including sales time and expense chasing them to get a contract as well as support resources into your CAC?…

Yes, this. One of the big things I talk about with sales is the difference between changing a priority for a customer vs. adding distinct new things. As a recent example, a client wants better and faster feedback on the trial they're conducting (we're in the med-tech space), and we've already got a new dashboard designed and on our product roadmap. I'm more than happy to prioritize that over other product pieces if i…

We (I) feel these same things.

>when they ask for something off the roadmap

Then we also get side tracked and lose focus. Leadership and management expend too much energy trying to figure out what to do. Then they want estimates from the developers so they can figure out an estimated ROI. But they rarely seem to worry about the true income potential, focusing mostly on just the initial development cost.

Pursue it? Don't pursue it? If we do, how will we? Will we be >hiring contractors/outsources (expensive, high management overhead), hiring new engineers (risky to grow headcount on a whim), or redirecting resources to tasks that are likely to have both lower ROI and provide lower growth for the re-tasked engineer.

Then is it really surprising that this lack of focus and discipline trickles down to those doing the work and the work itself? Technical debt in the making. It starts at the top.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#260

Post by kernelbandwidth » Tue, Jan 17, 2017, 8:11 PM UTC

jrs235 wrote: post #259

kernelbandwidth wrote: post #254

Earlier quoted context omitted.

Yes, this. One of the big things I talk about with sales is the difference between changing a priority for a customer vs. adding distinct new things. As a recent example, a client wants better and faster feedback on the trial they're conducting (we're in the med-tech space), and we've already got a new dashboard designed and on our product roadmap. I'm more than happy to prioritize that over other product pieces if i…

We (I) feel these same things. >when they ask for something off the roadmap Then we also get side tracked and lose focus. Leadership and management expend too much energy trying to figure out what to do. Then they want estimates from the developers so they can figure out an estimated ROI. But they rarely seem to worry about the true income potential, focusing mostly on just the initial development cost. Pursue it? Do…

Absolutely. A brief story on tech debt from the top:

One of the more frustrating things I've experienced is when I got push-back for implementing more project management process (we have a very light process, but when I took over it was sticky-notes-on-the-desk level). The complaint was "we can't slow down development to do more process". Very through-the-looking-glass, as I, the Engineer, was arguing for more management process and Leadership wanted less.

But of course, accurate estimates were needed, just, you know, without making measurements. I implemented some process anyway. We actually increased development speed from less churn and lowered communicated (consult docs before breaking someone's flow), improved estimates, and we've been able to better contain our tech debt.