Live data from Hacker News

Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

news.ycombinator.com

61–70 of 77 posts

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#61

I think you're probably asking about mistakes like deleting a database? But more often companies, orgs, or teams dying has to do with engineering management in my experience. I'll name a few I have experienced and you all can decide whether or not these count as "engineering mistakes?" - Saying nothing about it, but "secretly" moving the team to another country by changing the manager and director, and not hiring any…

> I figured out that the plan was to let natural attrition take over and for every person who quits from here, hire their replacement there.

Honestly, if the company has decided to move, that seems like a good way to do it, no? Nobody gets laid off, they avoid the bad will of axing a team outright or asking you to train your replacements; it becomes a simple gradual turn-over.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#63

Well, it didn't end the company but only because the company was bought and bought again - then killed. The new head of engineering had no experience in software. He developed a dislike for the language the product had originally been built in for his own reasons. He ordered a complete rewrite of the software in a different language. He hired contractors since we had no in house skills in the new language. This lack…

> He ordered a complete rewrite of the software in a different language. He hired contractors since we had no in house skills in the new language.

Ignoring for a moment the rest of it: What was the end state supposed to be? Like, pretend the rewrite went perfectly - now what? You still have nobody in-house that knows the language, let alone being familiar with the codebase!

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#64
post #59

Well, it didn't end the company but only because the company was bought and bought again - then killed. The new head of engineering had no experience in software. He developed a dislike for the language the product had originally been built in for his own reasons. He ordered a complete rewrite of the software in a different language. He hired contractors since we had no in house skills in the new language. This lack…

Out of curiosity: What was the old language and what was its replacement?

old = ruby (rails). New = Java.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#65
post #12

To protect their identity I won't go into specifics, but not implementing anti-tampering on local and remote backups i.e. protection from root. Backups only residing on live systems. Not protecting systems from bad automation. Not deprecating old automation frameworks and continually adding new automation frameworks. I am intentionally excluding specific incidents.

How did not doing these things sink the company? The problems stated are bad, but did something actually happen as a result, like a rogue script deleting things and the company no longer in business?

I think the implication of the sum of the things listed is that an old backup system overwrote/mangled newer backups because nobody turned it off and the two systems were targeting the same storage.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#66
post #5

I think you're probably asking about mistakes like deleting a database? But more often companies, orgs, or teams dying has to do with engineering management in my experience. I'll name a few I have experienced and you all can decide whether or not these count as "engineering mistakes?" - Saying nothing about it, but "secretly" moving the team to another country by changing the manager and director, and not hiring any…

I didn’t mean simple mistakes, but failures at design/architecture level (I was thinking more about technical leadership than management/staffing). This is a good list though!

Ah! You mean like adopting service buses for everything and watching them lose messages consistely. :)

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#67

I think you're probably asking about mistakes like deleting a database? But more often companies, orgs, or teams dying has to do with engineering management in my experience. I'll name a few I have experienced and you all can decide whether or not these count as "engineering mistakes?" - Saying nothing about it, but "secretly" moving the team to another country by changing the manager and director, and not hiring any…

> I figured out that the plan was to let natural attrition take over and for every person who quits from here, hire their replacement there. Honestly, if the company has decided to move, that seems like a good way to do it, no? Nobody gets laid off, they avoid the bad will of axing a team outright or asking you to train your replacements; it becomes a simple gradual turn-over.

Yeah, so in hindsight I feel that there's a better way to resolve this than the two options of "don't tell anyone" or "tell everyone."

The way I would do it, is tell people that we need to move their team to {{other country}} and therefore we would like to offer them either a severance package to find a new job _or_ work out a plan for them to transition to some other team that is in their location.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#68

I think you're probably asking about mistakes like deleting a database? But more often companies, orgs, or teams dying has to do with engineering management in my experience. I'll name a few I have experienced and you all can decide whether or not these count as "engineering mistakes?" - Saying nothing about it, but "secretly" moving the team to another country by changing the manager and director, and not hiring any…

>But more often companies, orgs, or teams dying has to do with engineering management in my experience.

Nothing can 10X sink a product faster than bad engineering management.

I once joined an established Enterprise software company where leadership had decided they wanted to make a new product. A lot of the justification was they saw the need with existing customers, and they could make $200M+ ARR by simply getting their existing customers to use it. They invested a lot of money in marketing, created a lot of hype, and built up a huge worldwide org.

One of the premises of the product was that if you had a fleet of machines using this product, it would be a force multiplier for extracting value out of a particular broadly defined use case.

Except, engineering and product management was so busy getting the engineers to build something that could scale to infinity, they never bothered to improve the basic stuff that let somebody get value during a trial period. Customers would struggle to deploy and integrate this thing onto one machine. And when they did, they didn't see the value.

Leadership always parroted the same thing. Slides about once you have thousands of machines using this, the value is there. But over 3 years nobody really implemented a nice way of integrating 5 machines, let alone 1000.

3 years later the org went from 3000 employees worldwide to a few sales people in every region and development mostly being done in India. Comically enough, the lack of financial resources meant that the new management had to focus on things that actually mattered to customers. So the product actually got a lot better. But by that time all the momentum and trust was gone.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#69

To protect their identity I won't go into specifics, but not implementing anti-tampering on local and remote backups i.e. protection from root. Backups only residing on live systems. Not protecting systems from bad automation. Not deprecating old automation frameworks and continually adding new automation frameworks. I am intentionally excluding specific incidents.

> but not implementing anti-tampering on local and remote backups i.e. protection from root. How do you accomplish this? I have some ideas/assumptions, but I'd like to hear more about the best practices here.

An easy way is to stream the backups to tape, and have someone swap the tapes. Typically people who do this take the recorded tapes offsite, but to prevent against an electronic attack it’s adequate to have the removed tapes even in the same room, just not in the tape drive.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#70

I know of a company that was pretty successful with a VB6 app they wrote in the late 90s, but twenty years later they needed to modernize. They spent two years on a from scratch rewrite that was never completed (second system syndrome). The company abandoned the rewrite, acquired a competitor, and rebranded it as the company's new product. It turns out the competitor's code base had serious deficiencies (couldn't sca…

Was this pre VNC or TeamViewer? Or did the folks in IT suddenly forget Remote Desktop was a thing?
Post reply on HN