Live data from Hacker News

Ask HN: Have you ever worked on a product that was killed by technical debt?

news.ycombinator.com

151–160 of 331 posts

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#151
post #83

I've been on one lack of loose coupling project. "All you're doing is importing from here and exporting to there, no problem, right?" and eighteen months later its somehow expanded into touching every part of the business except the vending machine firmware (no, was not a vending machine company). This leads to paralysis where touching anything makes anything and everything randomly crash. This was fundamentally a bu…

Until you said "vending machine" I thought you worked at my employer, as we've got one of those tar baby projects going on right now.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#152

Earlier quoted context omitted.

Do you mean Excel to CSV? Or a CSV importer? I totally agree about Excel importing, but CSV is trivial, no? Here is an Erlang version I happened to write yesterday: lists:map( fun(Row) -> string:tokens(Row, [SepChar]) end, string:tokens(InputStr, "\n") ). EDIT: I know this version won't support escaped separator/newline characters, but I made it for a specific use case in which I knew that would not occur. Adding tha…

Your edit is delightful. "No, honest guys, I knew CSV was more complicated. I just didn't need to make my code safe." Here's a csv parser in Erlang that actually attempts all that trivial stuff: https://github.com/rcouch/ecsv/blob/master/src/ecsv_parser.e... That's a lot more code than yours. And the notes even say it's not tolerant of badly formed CSVs.

I submitted that edit before the post had any replies.

Also, I did try to make clear that the given code was created 'for a specific use case in which I knew' that the format of the input files was tightly defined.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#153
Yes. I've been on projects that I believe would have succeeded if we'd approached them differently, with less debt. One was due to magpie syndrome - influential architects who enjoyed working with bleeding edge technology and convinced management not to listen to the more conservative recommendations of the developers who would be doing the majority of the work. The software was a wreck, and morale dropped (and the original architects left the team). I think the project would have worked with a better approach, but after a change in upper management, they decided to end the project completely.

Another was a rewrite of an application that had been very popular, but was written by someone who was actively learning on his own (a talented developer who later learned to write very clean and well organized code). It was, as I like to call it, "superglued" to the server. SQL statements that could have been handled with a join were instead handled by querying all the ids, running through them in a loop, building a new query, getting results, stashing them in an array, one by one. When those queries took too long, they were run in the background or just crashed the system. Changes were all made in place, on the prod system. There was no build, no archive, no nothing (this was in the early 2000s - even back then, this was a no-no, but it wasn't quite as shockingly unusual as it is now) I was part of a team that tried to rewrite it, but halfway through the organization was frustrated with the time and expense and terminated the project.

Interestingly, I think there's almost always a psychological or political factor in play when a project is killed purely by "Technical debt". In theory, debt itself shouldn't really play much of a role in whether a project is worth pursuing, because it's all sunk cost. If a project has a positive enough ROI that it would be worth pursuing as a greenfield project, then even the worst case scenario, trash everything and start over, is a net win.

However, I think what happens is that some people never wanted to see the project happen in the first place, or people get very frustrated with the expense, or people start to suspect that the failure was inevitable and that the technical problems are just a distraction. Think Jurassic Park (which in many ways is a story of a software project failure, a theme that is much stronger in the book). At the end, during the "post mortem", some of the characters think that the problem was in the approach. That with a bigger budget, less dependency on a few people, less corner cutting due to a lowball bid from a software developer, enforced through threats to reputation and lawsuits, that with a better approach it all would have worked. On the other hand, you have the chaotician, who insists from the start that a project like this will fail inevitably.

My guess is that if it was ever worth pursuing, it is always worth pursuing. The problem is, we can never really tell. Sometimes failure due to technical debt is taken as a sign that this failure was inevitable. Other times, it isn't.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#154
Absolutely. Tangent but...

I've long thought that "premature optimization is the root of all evil" is sort of a waste of breath... not because it's wrong but because over-engineering is a far greater problem today than premature optimization.

Over-engineering is a plague in modern software. Most of the failures due to technical debt that I've seen involved cases where "smart" developers built swiss army chainsaws to do things that required a hammer. This also often results in products that require orders of magnitude more resources than they should, which makes cloud vendors like Amazon and Digital Ocean a lot of money I guess.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#156

Earlier quoted context omitted.

Seen this a lot. A lot of companies think they are "product" companies, but due to their unwillingness to push back on customers, they become custom engineering shops, bolting on little one-off mods to their project over and over to appease bad customers (or to appease POTENTIAL customers who haven't even bought the product yet). Stop me when you recognize this one: "Hey your product is great, but we really want some…

"Let us use our logo" doesn't seem like an unreasonable request.

I remember my first day on the job too.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#157
post #109

There are multiple cases of "killed by technical debt". There's the case of mysterious and unsolvable breakage. The product simply stops working, and the team is unable to get it working again, period. This can happen with really ancient legacy products where the original team is gone, or young products that are written badly by inadequate teams. There's the case of unpleasantness. A product is so difficult and slow…

Some of the worst examples of that are when a project uses a custom build of a library of which the source no longer exists and no record of the changes exists either.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#158
post #157
post #109

There are multiple cases of "killed by technical debt". There's the case of mysterious and unsolvable breakage. The product simply stops working, and the team is unable to get it working again, period. This can happen with really ancient legacy products where the original team is gone, or young products that are written badly by inadequate teams. There's the case of unpleasantness. A product is so difficult and slow…

Some of the worst examples of that are when a project uses a custom build of a library of which the source no longer exists and no record of the changes exists either.

Reintegrating future changes in the upstream is also made nearly impossible as a result; our tech lead made a change to numpy a few years ago, didn't manage to get it accepted by the project, and we're stuck with this version until the sun burns out.

If there are changes in future numpy versions we want, it's up to us to backport them, which is nowhere near our core business.

There's a lot to be said for standardization and 'boring.'

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#159
post #94

Earlier quoted context omitted.

You need a lot more than that to handle CSV in the wild (quoting, Unicode, line termination, etc.) but the real killer I see is when it's edited by humans. The special cases for errors and inconsistencies will add up quickly; in some cases you may be able to reject invalid data but you may not have that option or an easy way to tell whether any particular value is wrong. Excel takes that, adds some fun things like pe…

That is very interesting, thanks! I hadn't thought about Unicode or tolerating human error. Although the times I have worked with it have been when it is a transport medium between two computer programs.

That's definitely a less-aggravating situation by far. I've had a lot of cases where a significant amount of specialist human time was in a spreadsheet and it's really made me wish there was an Excel-for-data which acknowledges how many people are using it for semi-structured data like this.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#160

Earlier quoted context omitted.

Template in one database table I can live with (pros and cons, multiple front-ends, etc). One template broken up in to 12 tables requiring an 100+ line SQL statement with concat()s and HTML interspersed is insane. Had there been an API or utilities with it to manage it, it might have been manageable, but nope - just "write some queries". Also, just repeated your comment to a friend who said "that's the worst thing yo…

(blown away by all the responses to my original question!!!) Your story here makes me laugh if only because of a very painfully familiar memory. Luckily this wasn't a big production system but rather an internal tool (that I guess clients did also use but it wasn't part of 'production' per se) that was written entirely in perl_cgi filled with cryptic regular expressions written in complete spaghetti code and it would…

We have a similar application in PHP. By the time I've traced through all of the included files that are touched by a particular function, I've forgotten what I'm looking for. It's truly a nightmare.
Post reply on HN