Live data from Hacker News

Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

news.ycombinator.com

1–10 of 77 posts

Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#1
See title - I think a collection of failure stories would be a useful learning resource.

Edit: I’m thinking primarily about technical decisions, though of course these are made in a wider context (eg choosing tech that’s impossible to hire for).

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#2
Saw someone buy a lot of bumper stickers to promote a political issue, without paying sufficient attention to the printing details. The ink was not UV tolerant. Used outside, the stickers became blank white within a week.

as i recall part of the slogan was "long term thinking"

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#3
I think you're probably asking about mistakes like deleting a database? But more often companies, orgs, or teams dying has to do with engineering management in my experience. I'll name a few I have experienced and you all can decide whether or not these count as "engineering mistakes?"

- Saying nothing about it, but "secretly" moving the team to another country by changing the manager and director, and not hiring anymore in this country, even when team members quit. It took awhile but I figured out that the plan was to let natural attrition take over and for every person who quits from here, hire their replacement there.

- Different company: after being acquired, the acquiring company communicated for eight months that no changes in staffing would take place, and that once the acquisition closes, everyone will be put on exciting new projects and a new bright future will emerge. Then, after the acquisition closed, laying off a third of the company immediately. Another third chose the exit soon after. Eventually, the company died.

- Third company: after being acquired by a large competitor, the director of technology for this third company promised them a new product for four years. I'm led to understand by those closer to the situation than myself that this director then proceeded to coast on vaporware demos for those four years, claiming the need to "pivot" or "reboot" the product as necessary, and promising more and more pie-in-the-sky fantasies until finally the gig was up and he was fired. That subsidiary also officially closed its doors eventually.

Just a few from my own experience!

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#4
To protect their identity I won't go into specifics, but not implementing anti-tampering on local and remote backups i.e. protection from root. Backups only residing on live systems. Not protecting systems from bad automation. Not deprecating old automation frameworks and continually adding new automation frameworks. I am intentionally excluding specific incidents.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#5

I think you're probably asking about mistakes like deleting a database? But more often companies, orgs, or teams dying has to do with engineering management in my experience. I'll name a few I have experienced and you all can decide whether or not these count as "engineering mistakes?" - Saying nothing about it, but "secretly" moving the team to another country by changing the manager and director, and not hiring any…

I didn’t mean simple mistakes, but failures at design/architecture level (I was thinking more about technical leadership than management/staffing). This is a good list though!

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#6

To protect their identity I won't go into specifics, but not implementing anti-tampering on local and remote backups i.e. protection from root. Backups only residing on live systems. Not protecting systems from bad automation. Not deprecating old automation frameworks and continually adding new automation frameworks. I am intentionally excluding specific incidents.

> but not implementing anti-tampering on local and remote backups i.e. protection from root.

How do you accomplish this? I have some ideas/assumptions, but I'd like to hear more about the best practices here.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#9

To protect their identity I won't go into specifics, but not implementing anti-tampering on local and remote backups i.e. protection from root. Backups only residing on live systems. Not protecting systems from bad automation. Not deprecating old automation frameworks and continually adding new automation frameworks. I am intentionally excluding specific incidents.

> but not implementing anti-tampering on local and remote backups i.e. protection from root. How do you accomplish this? I have some ideas/assumptions, but I'd like to hear more about the best practices here.

1st level of protection is by keeping your live production data & backup data in separate accounts.

(If 1 account gets compromised the other is still available)

Perhaps 2nd level is by using cloud specific features - such as AWS S3 Object locks (even owner can't delete the object till the retention period).

my 2cents

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#10
post #8

Ignoring the cloud for too long.

Been there, done that. They were bursting out of an ancient VMware cluster on private hardware in traditional hosting, with a residential router facing the internet. Hard yikes.
Post reply on HN