Knight Capital lost $465 million in 45 minutes caused (at least in part) by technical debt and poor development practices. Summary: http://pythonsweetness.tumblr.com/post/64740079543/how-to-lo...
I'd say it wasn't due to technical debt, more a start-up like development approach to a company that trades millions within seconds in full automation. It sounds like the deployment process wasn't that complicated for a company of that size, but it was deployed without a single check by a second person. If you're trading automatically, you'll need a very, very solid deployment and audit process, even if you're just a…
Ask HN: Have you ever worked on a product that was killed by technical debt?
111–120 of 331 posts
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#112Technical debt destroyed the team.
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#113Earlier quoted context omitted.
Do you mean Excel to CSV? Or a CSV importer? I totally agree about Excel importing, but CSV is trivial, no? Here is an Erlang version I happened to write yesterday: lists:map( fun(Row) -> string:tokens(Row, [SepChar]) end, string:tokens(InputStr, "\n") ). EDIT: I know this version won't support escaped separator/newline characters, but I made it for a specific use case in which I knew that would not occur. Adding tha…
You need a lot more than that to handle CSV in the wild (quoting, Unicode, line termination, etc.) but the real killer I see is when it's edited by humans. The special cases for errors and inconsistencies will add up quickly; in some cases you may be able to reject invalid data but you may not have that option or an easy way to tell whether any particular value is wrong. Excel takes that, adds some fun things like pe…
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#114I'm currently working on the reincarnation of a project that was killed by technical debt -- TWICE. The original codebase was about 20 years old. It was control code for something best described as an industrial robot. Written for the last 20 years by greybeards who knew a lot about the manufacturing process, and were reasonably good at getting a product out the door. But the whole thing was riddled with #ifdefs for…
Not to say I haven't seen this effect myself many times...
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#115Knight Capital lost $465 million in 45 minutes caused (at least in part) by technical debt and poor development practices. Summary: http://pythonsweetness.tumblr.com/post/64740079543/how-to-lo...
I'd say it wasn't due to technical debt, more a start-up like development approach to a company that trades millions within seconds in full automation. It sounds like the deployment process wasn't that complicated for a company of that size, but it was deployed without a single check by a second person. If you're trading automatically, you'll need a very, very solid deployment and audit process, even if you're just a…
In the enterprise, this is called "mature" and is a sign of great sophistication.
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#116Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#117I wrote a complicated and horrendously ugly scraping PHP script with hundreds of regexes to generate a PDF file from a CMS. It worked in about 90% of all articles, and for the rest only needing a few manual fixes in InDesign.
I always intended this as a stopgap measure till we implemented a clean document structure, of which both the web page and the PDF could be created. But that never happened. Then I left the company.
A few years later the company made a redesign of the website and totally scrapped the PDF feature.
However I still sometimes see these PDF pages out in the wild, mostly saved by the authors of the articles on their own web sites, because authors are allowed to link to their own content for free. It's a shame because I think these PDF versions of the articles were beautiful. But as you would say, technical debt killed the feature.
Another project for a NPO has been written in Python using an ancient back-end. When the shared hosting provider made an upgrade, the back-end died, and I was not able to repair it and I was not paid enough to invest too much time. I bluntly told the board that their project is as dead as a dodo, and they accepted that.
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#118Earlier quoted context omitted.
My understanding is that HVDC had advantages. That said, I was also intending that to include the distribution in your house.
HVDC does have advantages in certain scenarios (very long transmission lines, for example) but parent is still correct--the majority of the grid makes way more sense with AC.
Which, amusingly, is fitting for the tech debt debate. Eradicating some choices from the project is likely to be missing the point. Just as eradicating AC from all power would be short sighted/wrong.
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#119I built this extranet app for a Fortune-class / NYSE company in 2001. They were a Lotus Domino shop so for that and various other reasons the extranet was deployed in Domino. The initial rollout was considered quite successful, but it was definitely "v1" code, and I'm being really generous with the code quality. Plus, Domino.
The application was considered a stopgap until the shop had become fully Microsoft-centric, at which point it was expected to be migrated to .NET. That was expected to be in ~5 years.
The result was that no investment was made into the app for over fifteen years. Every now an then an enhancement would be needed, and a contractor would be called up to bolt on a feature in shockingly slipshod manner (this app is much too complex for the average Domino dev). But no technical debt was ever cleaned up, because "meh, we're going to replace that app by 2007."
2007 was 10 years ago. In the meantime two projects to replace the app were spun up and killed. The app is finally being retired this year. I was called up at the 11th hour to jump back in (15 years later) to help support the thing through the conversion, as the one existing Domino dev they had on staff finally (wisely) jumped ship.
I cannot even begin to describe the state of this app.... it's a case study in "how to not manage IT."
---
Another recent client was a content-creation shop (think glossy magazines). Their outgoing sr dev had deployed a CMS that nobody had (or has) ever heard of. This CMS was originally developed during the glory days of XML. Believe it or not, the app worked by loading all of the CMS content into a single in-memory XML document. This was probably OK for a brochure site, but this was a site with hundreds of thousands of pages of content. As a result the application required a server with 64GB of RAM just to launch. Also - launching the app took about ten minutes after the server OS was loaded. And there was no server farm, just the one server. If the app was ever stopped, it would stay down for at minimum 10 minutes.
I came in to fill in temporarily and to try to find someone to staff the position permanently. Even with a competitive salary, nobody qualified wanted the job.
Meanwhile, the same company also had a set of blogs that they managed in WordPress....
Re: Ask HN: Have you ever worked on a product that was killed by technical debt?
#120Earlier quoted context omitted.
You need a lot more than that to handle CSV in the wild (quoting, Unicode, line termination, etc.) but the real killer I see is when it's edited by humans. The special cases for errors and inconsistencies will add up quickly; in some cases you may be able to reject invalid data but you may not have that option or an easy way to tell whether any particular value is wrong. Excel takes that, adds some fun things like pe…
I know of at least one company whose entire business is handling this stuff. They find growing companies as they hit critical mass and need to move their Excel data into a real database. The product is just "Your data is hideous and was entered by hand without validation or formatting; it'll never convert and it'll be wrong when it does. We can help." They handle all kinds of theory and technical stuff, like normaliz…