Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 77 posts

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#31

Post by tevon »

gitfan86 wrote: post #22

Palantir is operating under the idea that "The AI will figure it out once we can access all the data" This is a mistake because: 1. A lot of the data at big orgs is garbage or only understandable within a certain context by specific people. 2. Internal politics within lots of organizations prevents access to this data. 3. The AI cannot just figure it out. You need tons of humans in the mix which brings you back to #1…

Have you ever touched Palantir software? This is pretty much the exact opposite of what their software does...

It lets the right people get access to, and make sense of, their companies shit data.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#32

Post by throwaway348473 »

gitfan86 wrote: post #22

Palantir is operating under the idea that "The AI will figure it out once we can access all the data" This is a mistake because: 1. A lot of the data at big orgs is garbage or only understandable within a certain context by specific people. 2. Internal politics within lots of organizations prevents access to this data. 3. The AI cannot just figure it out. You need tons of humans in the mix which brings you back to #1…

Word on the street is a couple years back Palantir knocked a good chunk of SOCOM's intelligence capabilities offline for a few weeks.

Intelligence analysts like to use Communities of Interest "COIs" to keep track of things. These are kind of like wikis. A COI on the Middle East may have groups like Al-Qaida and ISIS, and people like MBS, and countries like Saudi Arabia and links between them. They can also have hierarchies, like "planes" split into "prop" and "jet" which in turn splits into "military" and "civilian," think decision trees.

One problem with these is they can take a while stand up. COIs need at least a few hundred objects in them before they can be useful. Palantir came along and said, "well, we can help stand them up with our AI."

They did their work, apparently didn't properly test it and just did things in production (maybe SOCOM doesn't have a dev environment, who knows?), and when they pushed run it created hundreds of thousands of objects throughout the COIs, with levels in the hierarchy that didn't make any sense. Like, imagine "747" falls under "Helicopter" which falls under "Cessna" which is under the "Nation" category.

It took weeks for SOCOM to rebuild things. I'm not sure if they had problems with backups or maybe they didn't properly have things backed up.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#33

Post by paxys »

cyberge99 wrote: post #29

Not sure if it will sink the company, but moving from very good native apps to a single pwa/electron style app that was poorly written. The business justification seemly makes sense to have a single development train. But the execution was horrible. Bugs, horrible performance. Inconsistent features copied over from the various native apps, etc. It was released as a major upgrade, but it was almost alpha grade in real…

Why limit to native and pwa? Moving from a good app to a poorly written app will sink a product regardless of language or framework used.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#35

Post by blablabla123 »

Worked at a startup that built a 3D modelling tool in the browser. They did a major refactoring that was still ongoing. At this point in the core team of 5 they churned through 10 engineers in 1 year. So when I started my job I learned that you could neither create a working build nor would any of the tests run. (Not to speak about the CI pipeline) Eventually the build could be fixed as well as the few tests. But as it turned out every basic user functionality had to be rewritten and the structural refactoring was still in progress.

IMHO the refactoring sounded very reasonable since the old code base was unmanageable. But there was a culture of extreme pressure from the CTO, making everyone haste through tasks preventing everyone to do "proper engineering". At the time the CTO was in some sort of permanent absence already. Afterwards the team lead left because of burnout, I also left and later on I heard they closed down the company.

Actually I've seen similar destructive refactorings at other places. At one they also had a lot of subtle problems leading to many user complaints. It could be fixed but by then it was already too late.

IMHO refactorings are great but it's always necessary to keep regressions in check all the time and really understand the design decisions of existing code.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#36

Post by daenz »

h2odragon wrote: post #2

Saw someone buy a lot of bumper stickers to promote a political issue, without paying sufficient attention to the printing details. The ink was not UV tolerant. Used outside, the stickers became blank white within a week. as i recall part of the slogan was "long term thinking"

A troll company should print the opposite political values, in UV tolerant ink, underneath :)

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#37

Post by have_faith »

A colleague accidentally swapped every product photo on one premiere league football team shop website with a kit photo of another premiere league team. He pushed up the change (with his testing image...) and went home without realising until angry phone calls started coming in.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#38

Post by jamil7 »

slg wrote: post #30

I think this community and the tech world in general overvalues engineering and I say this as an engineer myself. In my experience engineering or technology rarely seem to be the reason for a company's success or failure. There are certainly outliers in which a tech is so incredible that it alone can build a company. There are also some engineering mistakes that can be too costly to fix later and people can always be…

I think this is generally true and agree 90% of the time although I've once seen the opposite in which what was otherwise a successful small SaaS severely underinvested in product. They had an MVP that they basically left unchanged for years and years while ramping up sales and marketing which worked for a while before competitors started popping up with better design/ux/perf/features etc.

Re: Ask HN: What (almost) company sinking engineering mistakes have you witnessed?

#40

Post by spiffytech »

I know of a company that was pretty successful with a VB6 app they wrote in the late 90s, but twenty years later they needed to modernize. They spent two years on a from scratch rewrite that was never completed (second system syndrome).

The company abandoned the rewrite, acquired a competitor, and rebranded it as the company's new product. It turns out the competitor's code base had serious deficiencies (couldn't scale to the company's size, had important bugs), and was very unpleasant to develop against. In the 6 months following the acquisition, well over half of the engineering team left the company (heavily skewed towards the highest-caliber engineers) because they couldn't stand what their jobs had become. Many long-time customers threatened to leave because of user facing technical issues in the acquired product.

A telling incident: the company needed to deploy an emergency fix to the mobile app. The only machine authorized to publish to the app store was a laptop that the owner of the acquired company had left at home overseas. To complete the deploy, he needed to call his wife and walk her through the publication process over the phone.

The company managed to fix enough of the problems that they're still around today, but for a while there was a lot of uncertainty around the company's future.

Post reply on HN