Live data from Hacker News

CrowdStrike Update: Windows Bluescreen and Boot Loops

old.reddit.com

531–540 of 1001 posts

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#531
post #349

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer startin…

It's a big stretch to call this the regulator's fault when its basic lack of testing by Microsoft and/or Crowdstrike. If a car manufacturer made safety belts that broke, you don't blame the regulators.

The root cause is automatic, mindless software update without proper testing - nothing to do with regulators.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#532
post #419

What I'm curious about: other than checkbox compliance, how does Crowdstrike convince companies to buy their product? Do they present evidence that their product is effective at protecting customers? Because certainly Crowdstrike customers still get hacked.

I've watched it occur countless times. Often times the people making the purchase decision are largely incompetent.

They usually come out and take your team to a nice lunch. Then they run you through a fancy slide deck and convince you to let them run some scaremongering reporting tool over your infra. By the end of the day, most of your leadership is convinced they need the solution.

Rinse and repeat hundreds of times and you have the 3rd party vendor hodgepodge hellscape that constitutes most large corporations' IT infrastructure.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#533
What do we do next week?

So assuming everyone uses sneaker-net to restart what’s looking like millions of windows boxes, there comes recriminations but then … what?

I think we need to look at minimum viable PC - certain things are protected more than others. Phones are a surprisingly good example - there is a core set of APIs and no fucker is ever allowed to do anything except through those. No matter how painful. At some point MSFT is going to enforce this the way Apple does. The EU court cases be damned.

For most tasks for most things it’s hard to suggest that an OS and a webbrowser are not the maximum needed.

We have been saying it for years - what I think we need is a manifesto for much smaller usable surface areas

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#535

Took down our entire emergency department as we were treating a heart attack. 911 down for our state too. Nowhere for people to be diverted to because the other nearby hospitals are down. Hard to imagine how many millions of not billions of dollars this one bad update caused.

And how many lifes lost?

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#538
Crowdstrike did this to our production linux fleet back on April 19th, and I've been dying to rant about it.

The short version was: we're a civic tech lab, so we have a bunch of different production websites made at different times on different infrastructure. We run Crowdstrike provided by our enterprise. Crowdstrike pushed an update on a Friday evening that was incompatible with up-to-date Debian stable. So we patched Debian as usual, everything was fine for a week, and then all of our servers across multiple websites and cloud hosts simultaneously hard crashed and refused to boot.

When we connected one of the disks to a new machine and checked the logs, Crowdstrike looked like a culprit, so we manually deleted it, the machine booted, tried reinstalling it and the machine immediately crashes again. OK, let's file a support ticket and get an engineer on the line.

Crowdstrike took a day to respond, and then asked for a bunch more proof (beyond the above) that it was their fault. They acknowledged the bug a day later, and weeks later had a root cause analysis that they didn't cover our scenario (Debian stable running version n-1, I think, which is a supported configuration) in their test matrix. In our own post mortem there was no real ability to prevent the same thing from happening again -- "we push software to your machines any time we want, whether or not it's urgent, without testing it" seems to be core to the model, particularly if you're a small IT part of a large enterprise. What they're selling to the enterprise is exactly that they'll do that.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#539

What do we do next week ? So assuming everyone uses sneaker-net to restart what’s looking like millions of windows boxes, there comes recriminations but then … what? I think we need to look at minimum viable PC - certain things are protected more than others. Phones are a surprisingly good example - there is a core set of APIs and no fucker is ever allowed to do anything except through those. No matter how painful. A…

Isn't that basically the point of WinRT and Windows 10 S Mode? The problem is getting developers to adopt the new more secure APIs.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#540
post #349

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer startin…

> My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer starting up

Hi fellow CVS employee. Are you enjoying your zscaler induced SSO outages every week that torpedo access to email and every internal application? Well now your VMs can bluescreen too. A few more vendor parasites and we'll be completely nonfunctional. Sit tight!

Post reply on HN