Ask HN: Have you ever worked on a product that was killed by technical debt? • Hacker News Board

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#101

Post by ryandrake » Tue, Jan 17, 2017, 3:38 PM UTC

Two examples I can recall. First time was an embedded application for a smallish (2-3 SW developers total in the company) firm. The code was, by my estimation, initially developed by a real talented senior guy. It at one time did what it was meant to do pretty well, and adhered to sound (at the time) best practices. Well Mr. Senior Guy ended up leaving for whatever reason, and since then, a string of less senior folks were brought in one by one to burn out maintaining the system throughout the years. And by "maintain" I mean "toss in every feature under the sun that the CEO asked for, whether or not it made sense and whether or not the rest of the system kept working". Every little thing that might get one potential customer to say yes was bolted on hastily and shipped as soon as it could compile. Just get it to work by any means and ship it. I was brought in as one of this long string of people. By the time I came, the CEO was frustrated that the system could not be added to quickly enough anymore, and that core functionality would fail in the field more and more often. It was a total mess, and eventually proved irredeemable. I'd like to say I left the system in better shape than I found it but at the end of the day it didn't matter. Technically, this project is not dead so it doesn't meet this Ask HN's criteria, but I don't think they ever have or will make another major release of it. It will limp along until they start taking technical debt seriously.

The second one was a mobile app that was originally ported from some legacy J2ME app and "gotten to work" on the iPhone platform. It was pretty much a straight port, data structure by data structure, from Java to Objective C, and didn't really use the platform properly at all. For example, each and every control was hand-crafted to mimic the original J2ME app, rather than using the built in UIs that iPhone provided. It got to the point where nobody could touch it without it falling over, and no senior person was willing to work on it anymore. I was senior enough in my career at that point that I could insist that a complete re-write was the only way to go. We did that successfully and the previous pile of technical debt was killed.

Those examples aside, I'd say that almost every place I have worked suffered from technical debt to a large degree. The common theme was a huge legacy code base that suffered for years (decades) from repeated "just cram it in and get it to work" abuse. The metaphor I always like to use is: No home builder on earth would, when their requirements were to build a 5 story apartment building, take a single story single family home and just add 4 floors. But seemingly every company building software attempts to do this.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#102

Post by Bartweiss » Tue, Jan 17, 2017, 3:39 PM UTC

acdha wrote: post #94

samwilliams wrote: post #90

Earlier quoted context omitted.

Do you mean Excel to CSV? Or a CSV importer? I totally agree about Excel importing, but CSV is trivial, no? Here is an Erlang version I happened to write yesterday: lists:map( fun(Row) -> string:tokens(Row, [SepChar]) end, string:tokens(InputStr, "\n") ). EDIT: I know this version won't support escaped separator/newline characters, but I made it for a specific use case in which I knew that would not occur. Adding tha…

You need a lot more than that to handle CSV in the wild (quoting, Unicode, line termination, etc.) but the real killer I see is when it's edited by humans. The special cases for errors and inconsistencies will add up quickly; in some cases you may be able to reject invalid data but you may not have that option or an easy way to tell whether any particular value is wrong. Excel takes that, adds some fun things like pe…

I know of at least one company whose entire business is handling this stuff. They find growing companies as they hit critical mass and need to move their Excel data into a real database. The product is just "Your data is hideous and was entered by hand without validation or formatting; it'll never convert and it'll be wrong when it does. We can help."

They handle all kinds of theory and technical stuff, like normalization and processing Excel-corrupted dates. But they also handle a lot of easy-but-agonizing tasks like regularizing single quotes into apostrophes, which crop as soon as you let humans enter free-form data.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#103

Post by bpyne » Tue, Jan 17, 2017, 3:40 PM UTC

Back in '96 I was in a startup company in the Boston area. We had a B2B app that we customized per customer. It worked over dial-up. We didn't foresee affordable internet connections coming. When it did we couldn't rewrite critical subsystems quickly enough.

Because our product was customized per customer - not just look-and-feel, we coded their business rules into it - we had problems scaling. Further, our design didn't lend itself to rapid development.

First our sales team left. Then developers started to leave. Within a couple of years after I left, the company folded. Many good experiences had in that company. Many lessons learned.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#104

Post by Bartweiss » Tue, Jan 17, 2017, 3:41 PM UTC

ryandrake wrote: post #101

Two examples I can recall. First time was an embedded application for a smallish (2-3 SW developers total in the company) firm. The code was, by my estimation, initially developed by a real talented senior guy. It at one time did what it was meant to do pretty well, and adhered to sound (at the time) best practices. Well Mr. Senior Guy ended up leaving for whatever reason, and since then, a string of less senior folk…

That's a great metaphor, thanks.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#105

Post by [deleted] » Tue, Jan 17, 2017, 3:41 PM UTC

taeric wrote: post #80

misja111 wrote: post #13

It happened to me twice. The first time was in a start-up at the beginning of the century, we were developing an electronic health record and we had outsourced the database abstraction layer to a company in Greece. In the beginning things went fine but after a while the development of the DAL went slower and slower and it became unstable as well. Eventually the word came out: the main developer of the DAL framework h…

This sounds less like technical debt, and more like liabilities of over engineering. Possibly feature creep. That is, technical debt is not necessarily tangled over-engineered code. It is more compromises that were made to actually ship and operate in the world. You can see this in the world with devices. Consider, technical debt is the reason you have AC delivered to your house going through as many converters as yo…

[deleted]

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#106

Post by nicostouch » Tue, Jan 17, 2017, 3:42 PM UTC

mgkimsal wrote: post #48

dx034 wrote: post #37

Earlier quoted context omitted.

Templates stored across a database is probably the worst thing I've seen repeatedly across projects. Just because a database can store everything doesn't mean it has to. Some people really seem(ed) to have an allergy to plain files for storage. A plain file with OS level caching will beat most (if not all) databases for static content. But doesn't sound as fancy, so it's probably harder to charge a lot of money for i…

Template in one database table I can live with (pros and cons, multiple front-ends, etc). One template broken up in to 12 tables requiring an 100+ line SQL statement with concat()s and HTML interspersed is insane. Had there been an API or utilities with it to manage it, it might have been manageable, but nope - just "write some queries". Also, just repeated your comment to a friend who said "that's the worst thing yo…

(blown away by all the responses to my original question!!!)

Your story here makes me laugh if only because of a very painfully familiar memory. Luckily this wasn't a big production system but rather an internal tool (that I guess clients did also use but it wasn't part of 'production' per se) that was written entirely in perl_cgi filled with cryptic regular expressions written in complete spaghetti code and it would concatenate together entire webpages that had bits of them rendered by including the contents of files strewn all over the file system and of course the logic to concatenate all the html together was strewn across a fistful of files which were in disparate locations. In short I was once asked to make a simple change to some html and after 5 days of reading through perl_cgi and developing a pure hatred for Larry Wall, I decided to do a java re-write that took 3 days. I mean... crikey. Haha.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#107

Post by ryandrake » Tue, Jan 17, 2017, 3:43 PM UTC

Bartweiss wrote: post #93

mgkimsal wrote: post #26

Yes and no. The project wasn't killed specifically because "you have technical debt". It was killed because there was no way for anyone to be effective with the combination of poor undocumented code. "We need to change the email message that goes out when someone registers". This took a team of (4?) people 5 calendar days to change. As a contractor, I had to vpn in to one system, then remote desktop over another vpn…

This is a great example of how technical debt 'kills'. It's not a murder, it's negligence and a slow demise. I went through one of these projects. The tech debt was never as bad as you describe, but it was a small company operating on a short runway. It also taught me an unfortunate lesson about non-technical founders and the dangers of outsourced code. The MVP for the company had been bought off the shelf. It worked…

Seen this a lot. A lot of companies think they are "product" companies, but due to their unwillingness to push back on customers, they become custom engineering shops, bolting on little one-off mods to their project over and over to appease bad customers (or to appease POTENTIAL customers who haven't even bought the product yet).

Stop me when you recognize this one: "Hey your product is great, but we really want something that does [totally different thing]. If you just add that thing, we will pay for all the NRE and you can sell it to others as part of your product! Win win!" Advice to junior developers: If you hear such talk in the hallway, RUN!

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#108

Post by masukomi » Tue, Jan 17, 2017, 3:43 PM UTC

lhnz wrote: post #15

Yes. Technical debt rendered it difficult to create features, and difficult to hire. The project dragged on as engineers came, tried to refactor and then left. I didn't stay. My understanding is that it was never released, so all of the money the company put into the project was wasted. This is maybe different from what people normally consider 'technical debt'. I don't mean just code aesthetics but also bugs, redund…

We have lots of tech debt (we are gradually paying it down), but we keep engineers for a loooong time (they basically don't leave unless they move) because of the social aspects of our team. A bunch of really nice, helpful people, who really want to make things better but are willing to balance "good" with "practical".

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#109

Post by beat » Tue, Jan 17, 2017, 3:48 PM UTC

There are multiple cases of "killed by technical debt".

There's the case of mysterious and unsolvable breakage. The product simply stops working, and the team is unable to get it working again, period. This can happen with really ancient legacy products where the original team is gone, or young products that are written badly by inadequate teams.

There's the case of unpleasantness. A product is so difficult and slow to work on that the company simply loses interest in it, and shuts it down rather than suffering through more maintenance. This does not happen with products that are highly successful business-wise, no matter how bad the suffering, so it's really a business failure rather than a technical one.

There's real antiquation. The product is dependent on a product of an outside vendor that is no longer available/maintained. I've dealt with this on a mainframe replacement, and it was horrible. I've also dealt with this in Java, and it was plenty painful there too.

And finally, there's replacement. A product is replaced (or intended to be replaced) by a new product that does more or less the same thing, only this time with a smart new team, in a hip new language, and by the gods, this time it's not going to be stupid and suck like that piece of crap the morons on the old team built! Most of these projects fail before they ever replace the old, working code, so I'm not sure this counts as technical debt failure.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#110

Post by napoleond » Tue, Jan 17, 2017, 3:51 PM UTC

taeric wrote: post #89

CydeWeys wrote: post #88

Earlier quoted context omitted.

> Consider, technical debt is the reason you have AC delivered to your house going through as many converters as you do devices. Often to the same target power characteristics for those devices. Bad example. AC power has many desirable characteristics for the local transmission grid. If you were to do the grid over from scratch you'd still use AC. You're also too focused on household electronic usage, which is a very…

My understanding is that HVDC had advantages. That said, I was also intending that to include the distribution in your house.

HVDC does have advantages in certain scenarios (very long transmission lines, for example) but parent is still correct--the majority of the grid makes way more sense with AC.