Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike Update: Windows Bluescreen and Boot Loops

old.reddit.com

Post reply on HN
Print view View original

361–370 of 1001 posts

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#361

Post by HenryBemis »

openrisk wrote: post #196

The details (the particular companies / systems etc) of this global incident don't really matter. When the entire society and economy are being digitized AND that digitisation is controlled and passes through a handful of choke points its an invitation to major disaster. It is risk management 101, never put all your digital eggs in one (or even a few) baskets. The love affair with oligopoly, cornered markets and powe…

> The details (the particular companies / systems etc) of this global incident don't really matter.

But they do matter. This is elementary. It's like saying "playing with matches doesn't matter". This is a problem that has happened before, albeit on smaller scale, and the solution/cure is well known and imho it should be established 2 decades ago to every org on the planet.

This is basic COBIT (or BYOFramework) stuff from 10-15-20 years ago.

How can you push a patch/update without testing it fist? I get it if you are a tiny company with 1 IT person an 20 local PCs. Stuff like that cripples you for a couple of days. But when you are an org, with 10k+ laptops, 500+ servers (half of them MS Win), how can you NOT test each and every update?

If you don't want to have the test/staging environments, then at least wait 1-3-5 days to see what the updates will do to others/the news.

Sorry not sorry guys and gals. I've been auditing systems and procedures for so many years, that this is a basic failure. "One cannot just push an update without testing it first" any update, no matter how small/innocent.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#362

Post by dschuetz »

aenis wrote: post #80

We are a major CS client, with 50k windows-based endpoints or so. All down. There exists a workaround but CS does not make it clear whether this means running without protection or not. (The workaround does get the windows boxes unstuck from the boot loop, but they do appear offline in the CS host management console - which of course may have many reasons).

Are you rolling out CS updates as is everywhere? Are you not testing any published updates immediately at least with some N-1 staging involved?

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#363

Post by templ2 »

scriptsmith wrote: post #3

It's crowdstrike: https://www.reddit.com/r/crowdstrike/comments/1e6vmkf/bsod_e... > 7/18/24 10:20PT - Hello everyone - We have widespread reports of BSODs on windows hosts, occurring on multiple sensor versions. Investigating cause. TA will be published shortly. Pinned thread. > SCOPE: EU-1, US-1, US-2 and US-GOV-1 > Edit 10:36PT - TA posted: https://supportportal.crowdstrike.com/s/article/Tech-Alert-W... > Edit 11:2…

[dead]

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#366

Post by dschuetz »

dschuetz wrote: post #355

I wonder who exactly messed up the update, microsoft or crowdstrike. Usually, there is pre-rollout update testing AND some companies use N-1 version staging for critical/production systems. For me it feels much more complex a failure than just "it's crowdstrike's fault". Everybody involved must have done something wrong.

That was quick: Microsoft is blaming "3rd party" and announced that "a fix is forthcoming". Very curious indeed. https://techcrunch.com/2024/07/19/banks-airlines-brokerage-h...

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#367

Post by SuperSandro2000 »

zmmmmm wrote: post #349

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer startin…

Sounds familiar

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#368

Post by tempaway4575144 »

ThePhysicist wrote: post #83

If it's true that a bad patch was the reason for this I assume someone, or multiple people, will have a really bad day today. Makes me wonder what kind of testing they have in place for patches like this, normally I wouldn't expect something to go out immediately to all clients but rather a gradual rollout. But who knows, Microsoft keeps their master keys on a USB stick while selling cloud HSM so maybe Crowdstrike ju…

Sounds like it was a 'channel file' which I think is akin to an av definition file that caused the problem rather than an actual software change. So they must have had a bug lurking in their kernel driver which was uncovered by a particular channel file. Still, seems like someone skipped some testing.

https://x.com/George_Kurtz/status/1814235001745027317

https://x.com/brody_n77/status/1814185935476863321

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#370

Post by grahar64 »

I wonder what the rollout procedure is for CrowdStrike. I put $100 down that this was a minor update they decided was so minimal it didn't need extensive testing.

So many places use the "emergency break glass rollout procedure" on every deploy because it doesn't require all the hassle

Post reply on HN