Live data from Hacker News

Ask HN: Have you ever worked on a product that was killed by technical debt?

news.ycombinator.com

111–120 of 331 posts

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#111
post #46
post #33

Knight Capital lost $465 million in 45 minutes caused (at least in part) by technical debt and poor development practices. Summary: http://pythonsweetness.tumblr.com/post/64740079543/how-to-lo...

I'd say it wasn't due to technical debt, more a start-up like development approach to a company that trades millions within seconds in full automation. It sounds like the deployment process wasn't that complicated for a company of that size, but it was deployed without a single check by a second person. If you're trading automatically, you'll need a very, very solid deployment and audit process, even if you're just a…

Call it "process debt", or "management debt" (i.e. the lack of investment in proper management and the culture that goes a long with it -- in favor of a "STFU and just add that feature now! I need it yesterday!" mentality). Either way, part of the same boat, basically.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#112
I've seen projects die not because of the technical debt existing, but because of lack of addressing it. The developers got sick of working around the debt to do trivial things, they left. Customer support personnel left because of the constant manual "tweaking" they had to do due to the technical debt. Management focused on tacking on features instead of paying off any technical debt which was having rippling effects throughout the company.

Technical debt destroyed the team.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#113
post #94

Earlier quoted context omitted.

Do you mean Excel to CSV? Or a CSV importer? I totally agree about Excel importing, but CSV is trivial, no? Here is an Erlang version I happened to write yesterday: lists:map( fun(Row) -> string:tokens(Row, [SepChar]) end, string:tokens(InputStr, "\n") ). EDIT: I know this version won't support escaped separator/newline characters, but I made it for a specific use case in which I knew that would not occur. Adding tha…

You need a lot more than that to handle CSV in the wild (quoting, Unicode, line termination, etc.) but the real killer I see is when it's edited by humans. The special cases for errors and inconsistencies will add up quickly; in some cases you may be able to reject invalid data but you may not have that option or an easy way to tell whether any particular value is wrong. Excel takes that, adds some fun things like pe…

Let's not forget Japan Post's CSV for all the Japanese Address data that contains some lines that are line-wrapped, that is, one record spans two or more lines in the CSV file. A line-wrapped CSV... I just can't even.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#114

I'm currently working on the reincarnation of a project that was killed by technical debt -- TWICE. The original codebase was about 20 years old. It was control code for something best described as an industrial robot. Written for the last 20 years by greybeards who knew a lot about the manufacturing process, and were reasonably good at getting a product out the door. But the whole thing was riddled with #ifdefs for…

Sounds like the software managers were not on top of things, otherwise how would they have allowed this type of design to be implemented?

Not to say I haven't seen this effect myself many times...

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#115
post #46
post #33

Knight Capital lost $465 million in 45 minutes caused (at least in part) by technical debt and poor development practices. Summary: http://pythonsweetness.tumblr.com/post/64740079543/how-to-lo...

I'd say it wasn't due to technical debt, more a start-up like development approach to a company that trades millions within seconds in full automation. It sounds like the deployment process wasn't that complicated for a company of that size, but it was deployed without a single check by a second person. If you're trading automatically, you'll need a very, very solid deployment and audit process, even if you're just a…

That's not a startup approach. That's an enterprise approach. Believe me, I've fought tons of resistance in automating deployment operations in the enterprise. There's a perception that automation is dangerous, and you need human checkpoints. In practice, I've worked on projects much, much larger than Knight Capital, where the deployment process was driven by a huge spreadsheet, and orchestrated by non-technical overseers telling techs what commands to run based on the spreadsheet in front of them. It's incredibly vulnerable to human error like "Oops, forgot to deploy to one of the eight servers in the cluster".

In the enterprise, this is called "mature" and is a sign of great sophistication.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#116
I am right now, but I can't take a step back and fix any of it because that wouldn't be "Agile". The project won't be declared a failure though, they will just keep adding more and more "resources" aka people to get the same amount of work done.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#117
Yes.

I wrote a complicated and horrendously ugly scraping PHP script with hundreds of regexes to generate a PDF file from a CMS. It worked in about 90% of all articles, and for the rest only needing a few manual fixes in InDesign.

I always intended this as a stopgap measure till we implemented a clean document structure, of which both the web page and the PDF could be created. But that never happened. Then I left the company.

A few years later the company made a redesign of the website and totally scrapped the PDF feature.

However I still sometimes see these PDF pages out in the wild, mostly saved by the authors of the articles on their own web sites, because authors are allowed to link to their own content for free. It's a shame because I think these PDF versions of the articles were beautiful. But as you would say, technical debt killed the feature.

Another project for a NPO has been written in Python using an ancient back-end. When the shared hosting provider made an upgrade, the back-end died, and I was not able to repair it and I was not paid enough to invest too much time. I bluntly told the board that their project is as dead as a dodo, and they accepted that.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#118
post #89

Earlier quoted context omitted.

My understanding is that HVDC had advantages. That said, I was also intending that to include the distribution in your house.

HVDC does have advantages in certain scenarios (very long transmission lines, for example) but parent is still correct--the majority of the grid makes way more sense with AC.

I meant my follow-on to be a concession, but worded it poorly. I thought it had advantages, but yes, I was thinking small appliance mainly. In particular, in home. And not just computers, but lights and control panels. Seems many things all use the same power characteristics and are now becoming complicated by dealing with AC.

Which, amusingly, is fitting for the tech debt debate. Eradicating some choices from the project is likely to be missing the point. Just as eradicating AC from all power would be short sighted/wrong.

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#119
I am working on a project now that bears some study.

I built this extranet app for a Fortune-class / NYSE company in 2001. They were a Lotus Domino shop so for that and various other reasons the extranet was deployed in Domino. The initial rollout was considered quite successful, but it was definitely "v1" code, and I'm being really generous with the code quality. Plus, Domino.

The application was considered a stopgap until the shop had become fully Microsoft-centric, at which point it was expected to be migrated to .NET. That was expected to be in ~5 years.

The result was that no investment was made into the app for over fifteen years. Every now an then an enhancement would be needed, and a contractor would be called up to bolt on a feature in shockingly slipshod manner (this app is much too complex for the average Domino dev). But no technical debt was ever cleaned up, because "meh, we're going to replace that app by 2007."

2007 was 10 years ago. In the meantime two projects to replace the app were spun up and killed. The app is finally being retired this year. I was called up at the 11th hour to jump back in (15 years later) to help support the thing through the conversion, as the one existing Domino dev they had on staff finally (wisely) jumped ship.

I cannot even begin to describe the state of this app.... it's a case study in "how to not manage IT."

---

Another recent client was a content-creation shop (think glossy magazines). Their outgoing sr dev had deployed a CMS that nobody had (or has) ever heard of. This CMS was originally developed during the glory days of XML. Believe it or not, the app worked by loading all of the CMS content into a single in-memory XML document. This was probably OK for a brochure site, but this was a site with hundreds of thousands of pages of content. As a result the application required a server with 64GB of RAM just to launch. Also - launching the app took about ten minutes after the server OS was loaded. And there was no server farm, just the one server. If the app was ever stopped, it would stay down for at minimum 10 minutes.

I came in to fill in temporarily and to try to find someone to staff the position permanently. Even with a competitive salary, nobody qualified wanted the job.

Meanwhile, the same company also had a set of blogs that they managed in WordPress....

Re: Ask HN: Have you ever worked on a product that was killed by technical debt?

#120
post #94

Earlier quoted context omitted.

You need a lot more than that to handle CSV in the wild (quoting, Unicode, line termination, etc.) but the real killer I see is when it's edited by humans. The special cases for errors and inconsistencies will add up quickly; in some cases you may be able to reject invalid data but you may not have that option or an easy way to tell whether any particular value is wrong. Excel takes that, adds some fun things like pe…

I know of at least one company whose entire business is handling this stuff. They find growing companies as they hit critical mass and need to move their Excel data into a real database. The product is just "Your data is hideous and was entered by hand without validation or formatting; it'll never convert and it'll be wrong when it does. We can help." They handle all kinds of theory and technical stuff, like normaliz…

Could you share the company name?
Post reply on HN