Were cloud providers (AWS and azure) so heavily impacted because they use CS internally or because so many users use CS?
CrowdStrike Update: Windows Bluescreen and Boot Loops
661–670 of 1001 posts
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#662I'm confused as to how this issue is so widespread in the first place. I'm unfamiliar with how Crowdstrike works, do organizations really have no control over when these updates occur? Why can't these airlines just apply the updates in dev first? Is it the organizations fault or does Crowdstrike just deliver updates like this and there's no control? If that's just how they do it, how do they get away with this?
I mean, they pay a lot of money to crowdstrike. A failure this widespread is a Crowdstrike dev issue.
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#663I want to see the internal postmortem of why this happened to CrowdStrike (if they are still in business)
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#664Do people not have test environments?
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#665This event is predicted in Sydney Dekker’s book “Drift into Failure”, which basically postulates that in order to prevent local failure we setup failure prevention systems that increase the complexity beyond our ability to handle, and introduce systemic failures that are global. It’s a sobering book to read if you ever thought we could make systems fault tolerant.
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#666Due to the scale I think it’s reasonable to state that in all likelihood many people have died because of this. Sure it might be hard to attribute single cases but statistically I would expect to see a general increase in probability.
I used to work at MS and didn’t like their 2:1 test to dev ratio or their 0:1 ratio either and wish they spent more work on verification and improved processes instead of relying on testing - especially their current test in production approach. They got sloppy and this was just a matter of time. And god I hate their forced updates, it’s a huge hole in the threat model, basically letting in children who like to play with matches.
My important stuff is basically air-gapped. There is a gateway but it’ll only accept incoming secure sockets with a pinned certificate and only a predefined in-house protocol on that socket. No other traffic allowed. The thing is designed to gracefully degrade with the idea that it’ll keep working unattended for decades, the software should basically work forever so long as equivalent replacement hardware could be found.
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#667Isn't Crowdstrike the same company the heavily lobbied to get make all their features a requirement for government computers? https://www.opensecrets.org/federal-lobbying/clients/summary... They have plenty of money for congress, but it seem little for any kind of reasonable software development practices. This isn't the first time crowdstrike has pushed system breaking changes.
Given its origin and involvement in these high profile cases I always thought Crowdstrike is a government subsidized company which barely has any real function or real product. I stand corrected I guess.
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#668Took down our entire emergency department as we were treating a heart attack. 911 down for our state too. Nowhere for people to be diverted to because the other nearby hospitals are down. Hard to imagine how many millions of not billions of dollars this one bad update caused.
> "Took down our entire emergency department as we were treating a heart attack."
Not questioning that it happened, but this was a boot loop after a content update. So if the computers were off and didn't get the update, and you booted them, they would be fine. And if they were on and you were using them, they wouldn't be rebooting, and it would be fine.
How did it happen that you were rebooting in the middle of treating a heart attack? [Edit: BSOD -> auto reboot]
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#669I work for a diesel truck repair facility and just locked up the doors after a 40 minute day :( .
- lifts wont operate.
- cant disarm the building alarms. (have been blaring nonstop...)
- cranes are all locked in standby/return/err.
- laser aligners are all offline.
- lathe hardware runs but controllers are all down.
- cant email suppliers.
- phones are all down.
- HVAC is also down for some reason (its getting hot in here.)
the police drove by and told us to close up for the day since we dont have 911 either.
alarms for the building are all offline/error so we chained things as best we could (might drive by a few times today.)
we dont know how many orders we have, we dont even know whos on schedule or if we will get paid.
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#670Crowdstrike did this to our production linux fleet back on April 19th, and I've been dying to rant about it. The short version was: we're a civic tech lab, so we have a bunch of different production websites made at different times on different infrastructure. We run Crowdstrike provided by our enterprise. Crowdstrike pushed an update on a Friday evening that was incompatible with up-to-date Debian stable. So we patc…
Interesting. How was the faulty upgrade distributed? Not from Debian archives I assume.