Running McAfee products on all production servers.
Lots of issues historically, and recently for that.
Ask HN: What (almost) company sinking engineering mistakes have you witnessed?
21–30 of 77 posts
Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?
#22Palantir is operating under the idea that "The AI will figure it out once we can access all the data"
This is a mistake because:
1. A lot of the data at big orgs is garbage or only understandable within a certain context by specific people. 2. Internal politics within lots of organizations prevents access to this data. 3. The AI cannot just figure it out. You need tons of humans in the mix which brings you back to #1 and #2.
Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?
#23Not engineering, but CEO banking the entire future of the company on a partnership, where we were dependent on the partner company for future financing, but the success metrics of the partnership were entirely in the hands of their sales department.
Never bet the future of your company on metrics that are entirely out of your control.
Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?
#24Friend used to work for Knight capital that basically went bankrupt by inadvertently hooking up their test trading system to the live market.
Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?
#25I have seen a couple of startups that wanted build and launch their product as fast as they can. The product was supposed to be a SAAS but in that rush they have chosen technologies that made adding features very difficult. While they were considering and procrastinating adding features, competitions started to pop up and started to take over the niche.
The lesson is that, if you are just planning to make an MVP make sure you have your scaling figured out. The blue ocean gets red faster than you can imagine.
Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?
#26I think you're probably asking about mistakes like deleting a database? But more often companies, orgs, or teams dying has to do with engineering management in my experience. I'll name a few I have experienced and you all can decide whether or not these count as "engineering mistakes?" - Saying nothing about it, but "secretly" moving the team to another country by changing the manager and director, and not hiring any…
Ironically, I'm going through the opposite of #1 - not a bad company, but decided it needed to move countries. Made a big announcement, said basically in 3 months you're all out the door, but keep up the hard work, and help with the transition! I think they _just_ hired their first couple of FTE's in the new location.
You can imagine the general reaction - many left, those with vacation time are taking it, progress ground to a halt, motivation and morale through the floor - so basically, burning three months of salary instead of doing exactly what you were bothered by - hiring new teams in the new locale, scaling down work in the old one, and then after things are stable in the new home, shutting down the old one.
It's truly baffling to me given that they're moving to a more expensive location, they're confused by higher costs of hiring, slowed progress and doubling of expenses.
End of the day it seemed like the best way to set several million dollars on fire - I imagine an executive saved their bacon somehow, but I really don't imagine there will be much of a bonus after the math is all done and tallied. I'm really curious how the new teams will react, realizing that there was this massive talent dump and nobody seems to know where anything came from - the transition is being handled as professionally as possible by those that remain, but I'm pretty sure I wouldn't take an FTE offer from them at this point given the track record.
Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?
#27Well, it didn't end the company but only because the company was bought and bought again - then killed.
The new head of engineering had no experience in software. He developed a dislike for the language the product had originally been built in for his own reasons. He ordered a complete rewrite of the software in a different language. He hired contractors since we had no in house skills in the new language.
This lack of in house skills also meant that oversight of the code the contractors produced was poor. Several of us raised alarms over this but we were ignored. Eventually, the day came when the first paying customer was signed for the new platform. It was a complete disaster. The code was very unstable and full of bugs. The deadline kept getting pushed farther and farther out.
The new owners concluded the entire division was a failure. They fired the head of engineering who had been in charge of the disaster and his boss too. Then they started work on a new version of the platform using entirely different people. The ones left behind on the old platform (in the old language) had nothing to do but provide support for the dwindling number of open contracts.
Millions of dollars went down the drain, years of time were wasted, customers were badly served and a lot of people were incredibly frustrated, all because one executive decided to ignore the in-house talent and experience and follow his own inflated ego instead.
Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?
#28Not limiting access to production.
It’s both an operational and technical decision because it results in inadequate tooling and specials.
A service I know well had an issue, and as a result they started doing what modern services do - nuking things going it will clear as clean versions come up.
But unknown to the ops team, they nuked a bunch of custom stuff that no one knew how to build or really what they contained. Developers with direct access to production had rolled them out.
7 days of partial outage later they covered this up by hiding parts of the app until they could straighten things out.
Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?
#29Not sure if it will sink the company, but moving from very good native apps to a single pwa/electron style app that was poorly written.
The business justification seemly makes sense to have a single development train. But the execution was horrible. Bugs, horrible performance. Inconsistent features copied over from the various native apps, etc.
It was released as a major upgrade, but it was almost alpha grade in reality.
That org probably saved a lot in engineering, but the customer backlash was immense and likely cost them more.
(It was a very popular and beloved note taking app. that tried to become an all-in-one day planner/calendar/todo/note platform).
Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?
#30I think this community and the tech world in general overvalues engineering and I say this as an engineer myself. In my experience engineering or technology rarely seem to be the reason for a company's success or failure. There are certainly outliers in which a tech is so incredible that it alone can build a company. There are also some engineering mistakes that can be too costly to fix later and people can always be negligent and not do something obvious like proper backups. However those situations are usually extremely rare. The most common engineering mistakes can be fixed with more money. Typically some other factors are what make or break a company. That is what leads to not having the money to fix those engineering mistakes.