Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 77 posts

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#51

Post by Jemaclus »

In my experience, most successful people (and by extension, people who run successful companies) don't really know why they were successful and often attribute it to their own actions (eg, I write great code), rather than luck or networking or having solid code reviewers or some external factor. Because of this, when they are successful, they often double down on whatever it is that they think they did the first time, and that frequently doesn't work a second time.

Maybe the first idea hit a niche that wasn't being satisfied by the market, and the second attempt tried to break into a heavily saturated market. Maybe they just lucked out by being in the right place at the right time, or having the right connection that could bring in a multi-million dollar contract, or...

From a broader perspective, I would say that the biggest and most common mistake that people make (engineers too!) is not to spend time examining the hows and whys of the success that you've had so far. Were you successful because you had brilliant ideas, or was it because you had a mediocre idea that filled an underserved niche market? Were you successful because you used Postgres or Ruby or Kafka or Elasticsearch? Were you successful because you created a culture of innovation and learning and team players? Or were you successful because you happened upon a fantastic solution for the specific problem at hand but can't generalize it to larger problems?

If you don't know why you were successful in the first place, it's hard to continue to be successful.

TL;DR: lack of introspection and evaluation of success criteria over time

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#52

Post by felizuno »

- Telling the engineers it's an MVP and then when they're done releasing and selling against it not even as beta, but as v1 production code. I see this all the time and I take a lot of money from these companies to rework and retrofit these systems to the quality they should have been before going live.

- Choosing load-bearing tech that you're unsure will meet 100% of your needs based on hype. For example there are tons of companies with marketing websites that for one reason or another can't have useful user analytics software attached or run A/B tests.

- Letting the engineers define the entire product. This leaves you with a "perfect" solution which then can't be explained let alone sold because the customer/user perspective was not properly considered. I've seen more than one innovative (and desperately needed) startup with patented tech fail this way despite having a groundbreaking solution at hand. Product design matters.

- Dividing your org into "good" teams and "bad" teams by funneling productive engineers towards important problems and not redistributing them once those problems are solved. This "good" team eventually spends all their time fixing the broken parts of the system that had been relegated to the "bad" teams (because they are so degraded that they are now the most important problems). This then causes those "good" engineers to quit because they don't like pure maintenance work, and the resulting rapid loss of knowledge cripples the business.

- Wasting large quantities of dev hours on things that won't ever make your cost of labor back. Obvious examples include companies that spend more to support IE than they bring in from IE users (I've observed this regularly doubling implementation times on a per-task basis).

- Native code avoidance. Everybody I know that has spent > 2y on a React Native project eventually switches to native code and wishes they had started that way. This is a sample size of 10+ real $MM projects. I've seen the same for many Electron-style apps. The resulting "stop the presses" rewrite is almost always started too late to save the day thanks to simple sunk cost fallacy.

... the list goes on an on. Statistics don't lie, the road to failure is wide and welcoming ;-)

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#55

Post by slg »

omegalulw wrote: post #42
slg wrote: post #30

I think this community and the tech world in general overvalues engineering and I say this as an engineer myself. In my experience engineering or technology rarely seem to be the reason for a company's success or failure. There are certainly outliers in which a tech is so incredible that it alone can build a company. There are also some engineering mistakes that can be too costly to fix later and people can always be…

This seems like such a short sided take. For a company to be labelled a "tech" company the tech itself should be the product. For example Apple, people buy iPhones and apple stuff because they perform well (e.g. fantastic camera, displays, M1...) and they work well with each other. Certainly at this point Apples name gets people to pay a premium but that's still grounded in the quality of their stuff. You can pick si…

I don't think that many people truly are choosing between an iPhone or Android phone based off the quality of the camera. I also don't think that Android phones are universally worse from an engineering perspective. Remember it wasn't that long ago that Apple designed a phone that lost reception if you held it the way that many people instinctually hold their phones. I think the differences between Apple and its competitors mostly comes down to different priorities and motivations. A product from a company that is committed to vertical integration is going to be vastly different than a product from a company that only exists to get you to use software which only exists to sell you ads.

Also Netflix didn't have a head start because of the quality of their streaming tech. They had a head start because of the business decision to invest in streaming before their competitors did. That early decision led to the quality of their library which is what attracted customers which is what allowed them to reinvest in their tech.

There are plenty of other examples. Twitter and Reddit come to mind as companies that had seemingly awful technology that would regularly fail. They succeeded through that. Companies like Facebook have certainly produced some good tech, but almost all of that has come after the company became a success. That is similar to Microsoft who got their big start selling someone else's tech.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#56

Post by felizuno »

coldcode wrote: post #44

In the early 2000's the consulting firm I worked for got a customer who was building a "revolutionary" (their words) new medical software suite. They were having difficulties shipping the product so we were hired to help. Turns out they were insanely paranoid about IP theft, so they had 20-30 programmers working on the code, each had zero access to any other programmer's slice (i.e. source code) of the app, only bein…

IMO Healthcare technology is the final boss of bad software orgs, and is the source of all ten of the top ten worst tech decisions I've ever seen.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#57

Post by cma »

mywittyname wrote: post #6
LinuxBender wrote: post #4

To protect their identity I won't go into specifics, but not implementing anti-tampering on local and remote backups i.e. protection from root. Backups only residing on live systems. Not protecting systems from bad automation. Not deprecating old automation frameworks and continually adding new automation frameworks. I am intentionally excluding specific incidents.

> but not implementing anti-tampering on local and remote backups i.e. protection from root. How do you accomplish this? I have some ideas/assumptions, but I'd like to hear more about the best practices here.

Maybe something like the write protect switch on LTO tape cartridges, so that if compromised you never overwrite the tape when trying to restore.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#58

Post by felizuno »

ohCh6zos wrote: post #53
k__ wrote: post #8

Ignoring the cloud for too long.

I've also seen companies ruin themselves and their competitive advantage in an attempt to go to the cloud.

I watched a startup implode going live with a $50k/mo AWS bill because their setup was too "future facing" (read: over-engineered for current scale)

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#59

Post by juanse »

irvingprime wrote: post #27

Well, it didn't end the company but only because the company was bought and bought again - then killed. The new head of engineering had no experience in software. He developed a dislike for the language the product had originally been built in for his own reasons. He ordered a complete rewrite of the software in a different language. He hired contractors since we had no in house skills in the new language. This lack…

Out of curiosity: What was the old language and what was its replacement?

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#60

Post by asychro1611 »

Company gave the Product team full reign and always "deferred" paying off tech debt to chase the next quarterly goal or business pivot. Eventually the tech stack was a giant, messy tarball with no test coverage and layers of hacks, but everyone could still deliver features because the original team was still there (this also provided justification to the Product team that tech debt payoff was unnecessary or could be "deferred" again).

After a while, business stagnation from constant pivots and new initiatives resulted in attrition and a downward spiral of budget cuts and reduced morale. Attrition and knowledge loss caused velocity to drop, which caused more of the team to leave, which made the business stagnation worse, which caused morale to go down more and budgets further cut, etc. Eventually the company couldn't recruit the same level of talent to replace people who left, hiring standards dropped dramatically, and it was now impossible to pay off tech debt or even really run the tarball reliably anymore (a Ruby monolith running an ancient, unsupported version of Rails with a million security holes and bugs).

Engineering leadership made a mass exodus, the few people that were left ended up on a death watch as the move to an outsourced engineering org from India was implemented on the way towards a full migration to a 3rd party vendor platform. Software engineering was completely eliminated from the company and the lesson the company leadership took from this was that "we never should have built the platform in the first place" along with a dose of "external business factors outside of our control caused the decline in revenue, forcing us to make hard decisions".

Post reply on HN