421–430 of 1001 posts
Took down our entire emergency department as we were treating a heart attack. 911 down for our state too. Nowhere for people to be diverted to because the other nearby hospitals are down. Hard to imagine how many millions of not billions of dollars this one bad update caused.
The thing that amazes me is how they've rolled out such a buggy change at such a scale. I would assume that for such critical systems, there would be a gradual rollout policy, so that not everything goes down at once.
So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer startin…
Great statement and one that needs to be seriously considered - would DORA regulation in the EU address this I wonder? Its a monster piece of tech legislation that SHOULD target this but WILL it - someone should use todays disaster and apply it to the regs to see if its fit for purpose.
Architecting technical systems is MUCH WAY easier than architecting social-economical systems. I hope one day all those tech-savvy web3 wannabe revolutionaries will start to do the real job a designing socially working systems, not only technically barely working cryptographically strong hamster-tapping scams
What I'm curious about: other than checkbox compliance, how does Crowdstrike convince companies to buy their product? Do they present evidence that their product is effective at protecting customers? Because certainly Crowdstrike customers still get hacked.
Of course, how they started from small was completely different.
i've seen photos of the bsod from an affected machine, the error code is `PAGE_FAULT_IN_NONPAGED_AREA`. here's some helpful takeaways from this incident: 1) mistakes in kernel-level drivers can and will crash the entire os 2) do not write kernel-level drivers 3) do not write kernel-level drivers 4) do not write kernel-level drivers 5) if you really need a kernel-level driver, do not write it in a memory unsafe langua…
Chances if Microsoft or Crowdstrike will be held liable for financial losses caused by this outage?
They all should have used some expensive corporate-and-government-level product that promises protection against exactly that kind of large scale attack on infrastructure.
So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer startin…
Removing the features that would allow sysadmins to actually do it automatically, even via the installer itself- would definitely be one way, but another one could be aggressive focus-stealing nags (similar to Windows' own nags) which in a server environment can actually cause some major issues, especially when automating processes in Windows (as you need to close the program when updating).
I think it's easy to blame the sysadmins, but I would also be remiss if I didn't point out that in the Windows world we have been slowly accepting these automatic dark patterns and alternative (more controlled) mechanisms have been removed over time.
I almost don't recognise the deployment environment today as to what it was in 2004; and yes, 20 years is a long time, but the total loss of control over what a computer is doing is only going to make issues like this significantly more common.