Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Have you ever worked on a product that was killed by technical debt?

news.ycombinator.com

Post reply on HN
Print view View original

191–200 of 331 posts

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#191

Post by PaulRobinson »

I was CTO of a company that had a two-week outage due to technical debt. I didn't sleep much for any of it. We fixed it, and we'd lost about 30% of our subscriber base in that period. The company took on new funding to survive, invested that in a new set of products, and shuttered the old stuff just to stay afloat.

I am currently working in a business where there is a nearly 8-year old Rails app (600+ models, 250+ controllers, 400+ libraries, LOC around 60k), that sits at the heart of everything we do.

The company is struggling to grow and believes the cause is that engineering is slow. We have asked to refactor this code base multiple times, and point to the technical debt as the cause features that should take a day to implement taking between 3-4 weeks, typically.

It is only recently that the penny has finally dropped and they've realised if they don't invest in replacing this thing (there is too much technical debt to fix, we're calling bankruptcy and moving to a brand new architecture piecemeal), the business is likely to fail within 1-2 years.

That means my current employer is likely to go bust because of technical debt within 2 years max unless we become really good at fixing this.

We are optimistic.

We have to be, right?

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#192

Post by rabidrat »

coredog64 wrote: post #131
Bartweiss wrote: post #102

Earlier quoted context omitted.

I know of at least one company whose entire business is handling this stuff. They find growing companies as they hit critical mass and need to move their Excel data into a real database. The product is just "Your data is hideous and was entered by hand without validation or formatting; it'll never convert and it'll be wrong when it does. We can help." They handle all kinds of theory and technical stuff, like normaliz…

I used to use Google Refine (now OpenRefine [0]) for this. It lets you load up the data and then apply rules to see if they are mostly correct. It doesn't get you all the way, but it is better than going blind on manually revising a huge Excel "database". [0] http://openrefine.org/

What do you use now?

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#193

Post by wuliwong »

I have never personally experienced a project failing because of technical debt. I have experienced "the opposite" of that where there was such a focus on doing things "the right way" it felt to me like development was slowed down by this. Our product iterations were slow and from my perspective it cost us valuable chances to test different product ideas.

I'm using this definition of technical debt.

  Technical debt is a concept in programming that reflects the extra development work that arises when code that is easy to implement in the short run is used instead of applying the best overall solution.[1]
[1] https://en.wikipedia.org/wiki/Technical_debt

##edit

The project I am referring to was a consumer web project. No mission critical type data.

In my experience (mostly consumer web, social media management for B2B) I have seen causes of failure to be heavily weighted towards product issues not engineering. The one big success I was a part of had the most technical debt. :) But that's probably because it lived the longest (and still lives today!). My experience is limited to being an employee at 4 different tech companies and several failed attempts of my own.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#194

Post by greenail »

nsxwolf wrote: post #168
beat wrote: post #109

There are multiple cases of "killed by technical debt". There's the case of mysterious and unsolvable breakage. The product simply stops working, and the team is unable to get it working again, period. This can happen with really ancient legacy products where the original team is gone, or young products that are written badly by inadequate teams. There's the case of unpleasantness. A product is so difficult and slow…

I worked for a company that grew considerably for 10 years and then lost its biggest client and folded quickly. We had spent a few years reworking our platform in a way that might have been successful enough to weather the storm of losing that client, but technical debt really slowed us down. Technical debt may not have killed the company directly, but we have to wonder how we might have done if we could have spent m…

This is revenue diversification not technical debt. Companies with a single customer funding the business should be actively pursuing a high priority strategy to reduce this risk.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#195

Post by rabidrat »

tylerpachal wrote: post #150
msluyter wrote: post #74

Earlier quoted context omitted.

Hey, sounds like we worked at the same place! That, or the "wedded to Oracle for life" is a common antipattern. I'd add "shared everything architecture" to the horrors. Yeah, once you get that deeply entrenched in Oracle, it's almost impossible to get away, and after that experience I vowed never to work at another Oracle shop.

Do you think the same thing could happen with cloud vendors like AWS?

Yes, and pretty easily if you buy into all the new features that no one else will ever have all of. If you stay clean with simple storage, compute, DB, email, then you should be okay.

Ideally you'd have some kind of plan though from the start, for which other cloud provider you would use and how the services would map, in case using AWS becomes untenable.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#196

Post by Bartweiss »

btown wrote: post #120
Bartweiss wrote: post #102

Earlier quoted context omitted.

I know of at least one company whose entire business is handling this stuff. They find growing companies as they hit critical mass and need to move their Excel data into a real database. The product is just "Your data is hideous and was entered by hand without validation or formatting; it'll never convert and it'll be wrong when it does. We can help." They handle all kinds of theory and technical stuff, like normaliz…

Could you share the company name?

I'll try to remember. I ran into them at a career fair a few years ago, so it's not leaping to mind, but it seemed like they had good software and a great market niche.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#197

Post by Anasufovic »

tylerpachal wrote: post #150
msluyter wrote: post #74

Earlier quoted context omitted.

Hey, sounds like we worked at the same place! That, or the "wedded to Oracle for life" is a common antipattern. I'd add "shared everything architecture" to the horrors. Yeah, once you get that deeply entrenched in Oracle, it's almost impossible to get away, and after that experience I vowed never to work at another Oracle shop.

Do you think the same thing could happen with cloud vendors like AWS?

Cloud product life cycles should definitely be more interesting. Azure for example already has a "classic" model and the new ARM model. Either way, avoid tightly coupling code with some external vendors service.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#198

Post by Glyptodon »

Do major design fails count as technical debt? Among other things I've run across an e-commerce site that required choosing a payment method before browsing products or putting things in the cart... (fortunately not my own job). (The reason for this odd choice was evidently that they wanted customers to have access to special offers/pricing/services/products depending depending on how they were going to pay. Cure worse than the disease type situation, IMO.)

I think I've worked for a company that did essentially mostly screw itself with technical debt, though.

They originally had overseas contractors write parts of their product without having a proper developer to assess and vet the results or requirements and it resulted in zero separation of concerns, business processes and display logic combined, and a terrible database structure. Eventually they wanted to move from their original layout and page design to a new one and it proved almost impossible without multiple developers spending around half a year. And even that was wasted effort because at the end of that they realized they wanted to actually have a mobile site and they still hadn't really created a good separation of concerns... Of course the whole time engineering and development wanted to refactor the business logic into separate code from the other layers, but management didn't want to expend resources on non-customer facing dev work.

I think the product still exists, but they've killed their momentum.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#199

Post by luhn »

PaulRobinson wrote: post #191

I was CTO of a company that had a two-week outage due to technical debt. I didn't sleep much for any of it. We fixed it, and we'd lost about 30% of our subscriber base in that period. The company took on new funding to survive, invested that in a new set of products, and shuttered the old stuff just to stay afloat. I am currently working in a business where there is a nearly 8-year old Rails app (600+ models, 250+ co…

Can you expand on how technical debt caused a two-week outage?

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#200

Post by TheCoreh »

luhn wrote: post #199
PaulRobinson wrote: post #191

I was CTO of a company that had a two-week outage due to technical debt. I didn't sleep much for any of it. We fixed it, and we'd lost about 30% of our subscriber base in that period. The company took on new funding to survive, invested that in a new set of products, and shuttered the old stuff just to stay afloat. I am currently working in a business where there is a nearly 8-year old Rails app (600+ models, 250+ co…

Can you expand on how technical debt caused a two-week outage?

If I had to guess, I'd say it's probably an issue with the build/deploy system. Perhaps someone deployed a broken build, then tried to revert/rollback, and realized that the previous version didn't build "cleanly" anymore.

This could happen if you have a lot of dependencies, switched compiler versions but left the binaries "in place" and deployed changes incrementally.

Post reply on HN