Live data from Hacker News

How CrowdStrike Stopped Everything

cacm.acm.org

11–20 of 28 posts

Re: How CrowdStrike Stopped Everything

#13
post #10

I know people many suffered with this and in some cases the suffering was real. For most people it was just an inconvenience. And I am a bit ashamed to say, the panic from users users who could not get to Office 365 or their Windows PC cannot boot still brings a bit of a smile to my face. I think there is a German word for this :) A relative worked from my house for those days because their internet was also broken.…

>I think there is a German word for this :)

That would be ‘schadenfreude.’

Re: How CrowdStrike Stopped Everything

#14
post #10

I know people many suffered with this and in some cases the suffering was real. For most people it was just an inconvenience. And I am a bit ashamed to say, the panic from users users who could not get to Office 365 or their Windows PC cannot boot still brings a bit of a smile to my face. I think there is a German word for this :) A relative worked from my house for those days because their internet was also broken.…

[deleted]

Re: How CrowdStrike Stopped Everything

#15
No real insights here, just high level explanation and empty quotes from folks.

> The root cause analysis (RCA) means that a CrowdStrike programmer(s) did not check their inputs before pushing an update to the CrowdStrike Falcon Windows Sensor in production.

How is it this isn’t just automated and / or that update automatically run in a VM or something and when it crashed the rollout prevented?

It’s not a new concept…

Re: How CrowdStrike Stopped Everything

#17
post #3

[flagged]

> When do we get rid of the traditional finance mental models from history like we are religious mental models from history? Customers need to stop buying from companies that are publicly listed. Publicly listed companies are designed for the next quarter - cutting corner is the name of the game. Also see: Boeing, GE, IBM, and recently Meta products that are basically unusable.

Private equity owned businesses are run in an even more predatory manner, not for next quarter's profits but just to extract every dime of equity as fees and services for their parent or as loan principal that can never be paid back.

And then venture-backed companies are ticking time bombs that abruptly close shop when they run out of money or are acquired, or at best themselves turn into the publicly traded companies you loathe.

"Good" companies are mostly just a consumer-friendly stage of the investment lifecycle.

Re: How CrowdStrike Stopped Everything

#18
post #15

No real insights here, just high level explanation and empty quotes from folks. > The root cause analysis (RCA) means that a CrowdStrike programmer(s) did not check their inputs before pushing an update to the CrowdStrike Falcon Windows Sensor in production. How is it this isn’t just automated and / or that update automatically run in a VM or something and when it crashed the rollout prevented? It’s not a new concept…

These sort of things happen occasionally despite all the possible safeguards you can set up. It's just gonna be something else you didn't predict and frankly the wrong thing to focus on.

The real problem this exposed is that somehow it's apparently the law for corporations to introduce a single point of failure into their entire IT system with absolutely zero fallback capacity or any workarounds at all. It's just entropy meeting stupidity.

It's like it was mandated for planes to have a built in off switch that can be triggered remotely and people then blame the company for accidentally triggering it, crashing every single one and killing everyone, instead of asking why the fuck do all planes have an off switch?!

Re: How CrowdStrike Stopped Everything

#19
post #7

Earlier quoted context omitted.

I don't really think it's possible to get rid of the finance mental models. Maybe momentarily at the best -- and that's why excellent engineers leave when "the company becomes too big". Everyone living in the real world has to think about his/her own financials and companies are obliged to do that too.

but there's the principal-agent problem ( https://www.investopedia.com/terms/p/principal-agent-problem... ) laying in there somewhere. The people making financial decisions are incentivized to minimize the financial cost (or maximize the gains), but only for some time period that's shorter than the life-time of the company, which, as far as the owners/shareholders are concerned, should be infinite. So this has an inh…

I don't think we disagree on this? Maybe I misread something...

Re: How CrowdStrike Stopped Everything

#20
post #15

No real insights here, just high level explanation and empty quotes from folks. > The root cause analysis (RCA) means that a CrowdStrike programmer(s) did not check their inputs before pushing an update to the CrowdStrike Falcon Windows Sensor in production. How is it this isn’t just automated and / or that update automatically run in a VM or something and when it crashed the rollout prevented? It’s not a new concept…

These sort of things happen occasionally despite all the possible safeguards you can set up. It's just gonna be something else you didn't predict and frankly the wrong thing to focus on. The real problem this exposed is that somehow it's apparently the law for corporations to introduce a single point of failure into their entire IT system with absolutely zero fallback capacity or any workarounds at all. It's just ent…

My understanding is this update crashed very consistently.
Post reply on HN