Live data from Hacker News

CrowdStrike Update: Windows Bluescreen and Boot Loops

old.reddit.com

631–640 of 1001 posts

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#631

Some Canonical guy I think many years ago mentioned this as their sales strategy a few year ago after a particularly nasty Windows outage: We don't ask customers to switch all systems from Windows to Ubuntu, but to consider moving maybe a third to Ubuntu so they won't sit completely helpless next time Windows fail spectacularly. While I see more and more Ubuntu systems, and recently have even spotted Landscape in the…

Seems like a reasonable strategy. Not just Ubuntu but some redundancy in some systems.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#632
post #80

We are a major CS client, with 50k windows-based endpoints or so. All down. There exists a workaround but CS does not make it clear whether this means running without protection or not. (The workaround does get the windows boxes unstuck from the boot loop, but they do appear offline in the CS host management console - which of course may have many reasons).

Do you need to manually fix all your windows boxes? Or is there a way to update it remotely?

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#636
post #613

I can't wait to see the CloudFlare traffic report after this. All those computers going down must have affected traffic worldwide. Even from Linux systems as their owners couldn't run jobs from their bricked Windows laptops.

It looks quite normal so far: https://radar.cloudflare.com/traffic

DE-CIX traffic is also often a good indicator during global events, looks normal: https://www.de-cix.net/en/locations/frankfurt/statistics

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#637
post #538

Crowdstrike did this to our production linux fleet back on April 19th, and I've been dying to rant about it. The short version was: we're a civic tech lab, so we have a bunch of different production websites made at different times on different infrastructure. We run Crowdstrike provided by our enterprise. Crowdstrike pushed an update on a Friday evening that was incompatible with up-to-date Debian stable. So we patc…

> we're a civic tech lab

Obviously not the point of your post, but say more? This sounds like it could be pretty cool!

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#638
Isn't Crowdstrike the same company the heavily lobbied to get make all their features a requirement for government computers? https://www.opensecrets.org/federal-lobbying/clients/summary... They have plenty of money for congress, but it seem little for any kind of reasonable software development practices. This isn't the first time crowdstrike has pushed system breaking changes.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#639

Took down our entire emergency department as we were treating a heart attack. 911 down for our state too. Nowhere for people to be diverted to because the other nearby hospitals are down. Hard to imagine how many millions of not billions of dollars this one bad update caused.

I remember a fed speaker in the 90s at Alexis hotel Defcon trying to rationalize their weirdly over-aggressive approach to enforcement by mentioning how hackers would potentially kill people in hospitals, fast forward to today and it's literally the "security" software vendor that's causing it.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#640
Wow, this hits close to home. Doing a page fault where you can't in the kernel is exactly what I did with my very first patch I submitted after I joined the Microsoft BitLocker team in 2009. I added a check on the driver initialization path and didn't annotate the code as non-paged because frankly I didn't know at the time that the Windows kernel was paged. All my kernel development experience up to that point was with Linux, which isn't paged.

BitLocker is a storage driver, so that code turned into a circular dependency. The attempt to page in the code resulted a call to that not-yet-paged-in code.

The reason I didn't catch it with local testing was because I never tried rebooting with BitLocker enabled on my dev box when I was working on that code. For everyone on the team that did have BitLocker enabled they got the BSOD when they rebooted. Even then the "blast radius" was only the BitLocker team with about 8 devs, since local changes were qualified at the team level before they were merged up the chain.

The controls in place not only protected Windows more generally, but they even protected the majority of the Windows development group. It blows my mind that a kernel driver with the level of proliferation in industry could make it out the door apparently without even the most basic level of qualification.

Post reply on HN