Live data from Hacker News

Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

news.ycombinator.com

41–50 of 77 posts

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#41
When I've seen companies sink for technology reasons, it's usually been because of the cumulative effect of tech debt accumulating over a couple years. Engineers need to hit a launch deadline, so they throw in some global variables or make a few private APIs public in the interest of expediency, figuring that they'll clean it up later. Later comes, and the engineers move onto new projects instead of cleaning up the previous mess. New code gets written that depends on the previous hacks. Eventually you end up with a big ball of mud that breaks whenever anybody touches it, and nobody can launch any new features. Now the company is forced into the position of rewriting everything from scratch, but can't do so strategically, they just need to do it now because they don't have any other options.

Companies failing for "big" single decisions - like rewriting their code from scratch, or a poor tech stack choice, or a poor initial architecture - are much more rare, for the simple reason that most experienced technical leadership knows that these are risky decisions and tends to put a lot of thought into them, and then there's a lot more organizational commitment to following through on the decision once it's been made. Also, if done from a position of market leadership, you can usually recover from them - many startups start with a poor tech stack and poor architecture, and just assume that they'll rewrite everything once they get lots of funding and a suitable lead over competitors. The insidiousness of the "creeping tech debt" scenario is that you often don't realize you're screwed until you've fallen behind the competitors, which means you enter the "rewrite and tech switch" scenario from the position of being market laggards, which can kill you.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#42
post #30

I think this community and the tech world in general overvalues engineering and I say this as an engineer myself. In my experience engineering or technology rarely seem to be the reason for a company's success or failure. There are certainly outliers in which a tech is so incredible that it alone can build a company. There are also some engineering mistakes that can be too costly to fix later and people can always be…

This seems like such a short sided take. For a company to be labelled a "tech" company the tech itself should be the product. For example Apple, people buy iPhones and apple stuff because they perform well (e.g. fantastic camera, displays, M1...) and they work well with each other. Certainly at this point Apples name gets people to pay a premium but that's still grounded in the quality of their stuff. You can pick similar examples across other tech companies too. For example, Netflix got their head start because of their streaming tech.

> The most common engineering mistakes can be fixed with more money.

That applies to pretty much all mistakes.

> Typically some other factors is what makes or breaks a company.

Tech is still pretty much at the top of that list, especially for "tech" companies.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#43

Friend used to work for Knight capital that basically went bankrupt by inadvertently hooking up their test trading system to the live market.

The SEC report [0] on this incident reads like a handbook of what not to do.

[0]: https://www.sec.gov/litigation/admin/2013/34-70694.pdf

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#44
In the early 2000's the consulting firm I worked for got a customer who was building a "revolutionary" (their words) new medical software suite. They were having difficulties shipping the product so we were hired to help. Turns out they were insanely paranoid about IP theft, so they had 20-30 programmers working on the code, each had zero access to any other programmer's slice (i.e. source code) of the app, only being allowed to use libraries and APIs to interact with. No one except the execs had access to the entire codebase, thus no code reviews, no unified architecture or design agreements; basically it was an app made up of 20-30 independent apps, all doing things differently without any coordination. After we got hired they fired all of their programmers and gave us the entire source code, it took months just to be able to build the app at all from source, but it was such a mess it was impossible to make something remotely shippable.

One day they vanished and owed us nearly $800,000. It was enough that our parent company just shut us down a few months later. Oddly enough we had a really good group of developers and likely could have rebuilt the whole app ourselves, but our parent insisted on trying to recover the money instead of just taking the IP.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#45
post #26

I think you're probably asking about mistakes like deleting a database? But more often companies, orgs, or teams dying has to do with engineering management in my experience. I'll name a few I have experienced and you all can decide whether or not these count as "engineering mistakes?" - Saying nothing about it, but "secretly" moving the team to another country by changing the manager and director, and not hiring any…

Ironically, I'm going through the opposite of #1 - not a bad company, but decided it needed to move countries. Made a big announcement, said basically in 3 months you're all out the door, but keep up the hard work, and help with the transition! I think they _just_ hired their first couple of FTE's in the new location. You can imagine the general reaction - many left, those with vacation time are taking it, progress g…

At least the company was being honest.

I have heard some companies hide the truth of relocating to another city and announce it right before the move. As employee, you either move with the company or find a new job.

It's probably better to build a team in the new location then announce the move in advance. It will be a shock either way.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#46

Earlier quoted context omitted.

> but not implementing anti-tampering on local and remote backups i.e. protection from root. How do you accomplish this? I have some ideas/assumptions, but I'd like to hear more about the best practices here.

Separate storage of file hashes would make tampering detectable. Actually preventing root modifying a file probably has to be done at the hardware level.

Heard of a case where an admin was bribed by competition to delete everything.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#48

Worked at a startup that built a 3D modelling tool in the browser. They did a major refactoring that was still ongoing. At this point in the core team of 5 they churned through 10 engineers in 1 year. So when I started my job I learned that you could neither create a working build nor would any of the tests run. (Not to speak about the CI pipeline) Eventually the build could be fixed as well as the few tests. But as…

This sounds more like bad management than bad engineering.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#49

Well, it didn't end the company but only because the company was bought and bought again - then killed. The new head of engineering had no experience in software. He developed a dislike for the language the product had originally been built in for his own reasons. He ordered a complete rewrite of the software in a different language. He hired contractors since we had no in house skills in the new language. This lack…

Amazing.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#50
I worked for a company that did an SAP modernization project. The IBM consultants did a large part of converting all the custom ABAP stuff. The idea was to get back to as vanilla SAP as possible and included a ton of Business Objects and data warehouse work as well to convert old reporting etc. They were constantly behind and decided to just push the load testing off the road map to hit the cio's arbitrary go live date. Within three days the data volume got large enough to grind the entire system to a halt and the company couldn't take, bill or fulfill orders. Of course the consultants were well out the door by that point. I spent months unwinding the stupid crap they did on the Business objects reporting side.
Post reply on HN