Live data from Hacker News

CrowdStrike Update: Windows Bluescreen and Boot Loops

old.reddit.com

501–510 of 1001 posts

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#501

The Windows ecosystem typically deployed in corporate PCs or workstations is often insecure, slow, and poorly implemented, resulting in ongoing issues visible to everyone. Examples include problems with malware, ransomware, and Windows botnets. In corporate environments, IT staff struggle to contain these issues using antivirus software, firewalls, and proxies. These security measures often slow down PCs significantl…

[deleted]

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#502

The Windows ecosystem typically deployed in corporate PCs or workstations is often insecure, slow, and poorly implemented, resulting in ongoing issues visible to everyone. Examples include problems with malware, ransomware, and Windows botnets. In corporate environments, IT staff struggle to contain these issues using antivirus software, firewalls, and proxies. These security measures often slow down PCs significantl…

> The Windows ecosystem typically deployed in corporate PCs or workstations is often insecure, slow, and poorly implemented

Yes, but that's not because of Windows itself (which is fast and secure out of the box) but because of an decades-old "security product" culture that insists on adding negative-value garbage like Crowdstrike and various anti-virus systems on the critical path, killing performance and harming real security.

It's a hard problem. No matter how good Windows itself gets and no matter how bad these "security products" become, Windows administrators are stuck in the same system of crappy incentives.

Decades of myth and superstition demand they perform rituals and make incantations they know harm system security, but they do them anyway, because fear and tradition.

It's no wonder that they see Linux and macOS as a way out. It's not that they're any better -- but they're different, and the difference gives IT people air cover for escaping from this suffocating "you must add security products" culture.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#503
post #349

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer startin…

>> It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it.

While orgs using auto update should reconsider, the fact that CrowdStrike don't test these updates on a small amount of live traffic (e.g. 1%) is a huge failure on their part. If they released to 1% of customers and waited even 24 hours before rolling out further this seems like it would have been caught and had minimal impact. You have to be pretty arrogant to just roll out updates to millions of customers devices in one fell swoop.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#504

Took down our entire emergency department as we were treating a heart attack. 911 down for our state too. Nowhere for people to be diverted to because the other nearby hospitals are down. Hard to imagine how many millions of not billions of dollars this one bad update caused.

i mean not just dollars but lives also right? do we have a way to track that?

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#505
post #349

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer startin…

That's some very twisted logic. If I expect someone to clean the kitchen as part of restaurant closeup checklist, and they fuck it all up, would I blame the checklist, or the person doing the work?

You blame the person fucking it up. In this case, it's someone who only cares about checking a box. Or someone who pushes broken shit.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#507
post #349

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer startin…

Consolidation / optimization of labor.

When Crowdstrike messes up and BSODs thousands of machines, they have a dedicated team of engineers working the problem and can deliver a solution.

When your company gets owned because you didn't check a compliance checkbox, it's on you to fix it (and you may not even currently have the talent to do so).

We see similar risk tradeoffs in cloud computing in general; yes, hosting your stuff on AWS leaves you vulnerable to AWS outages, but it's not like outages don't happen if you run your own iron. You're just going to have to dispatch someone a three hour drive away to the datacenter to fix it when they do.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#508

Lessons learned from this: - CS: Have a staging (production-like) environment for proper validation. It looks like CS has one of these bu they have just skipped it - IT Admins: Have controlled roll-outs, instead of doing everything in a single swoop. - CS: Fuzz test your configuration Anything I have missed?

It is possible Cloudflare did a timepointed release on this. Controlled roll-outs wouldn't work if all the daily chunked updates didn't activate the kernel driver until some point in the future.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#510

I guess this article might need some updating soon: https://www.crowdstrike.com/resources/reports/total-economic...

"Falcon Complete managed detection and response (MDR) delivers 403% ROI, zero breaches and zero hidden costs"
Post reply on HN