Live data from Hacker News

CrowdStrike Update: Windows Bluescreen and Boot Loops

old.reddit.com

471–480 of 1001 posts

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#471
post #349

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer startin…

Regulation or auditors rarely require specific solutions. It's the companies themselves that choose to achieve the goals by applying security like tinctures: "security solutions". The issue is that the tinctures are an approved remedy.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#472

IMO, having a mix of servers would help in mitigating issues like that. Like run stuff on Linux, windows and freebsd servers, so that you have OS redundancy should an issue affect one in particular (kernel or app). Just like you want more than a single server handling your traffic, you’d want 2 different base for those servers to avoid impacting them both with an update.

Not using this crap security software would mitigate this issue.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#473
Isn't a Windows BSOD the equivalent of a kernel panic? I don't understand how this is CrowdStrike's fault. Vanilla userspace operations shouldn't cause a kernel panic--that's a bug in the OS, not a bug in some user software. If anything, we should be blaming Windows here?

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#474
Lessons learned from this:

- CS: Have a staging (production-like) environment for proper validation. It looks like CS has one of these bu they have just skipped it - IT Admins: Have controlled roll-outs, instead of doing everything in a single swoop. - CS: Fuzz test your configuration

Anything I have missed?

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#477

So apparently "The issue has been identified, isolated and a fix has been deployed" https://x.com/George_Kurtz/status/1814235001745027317 Yet the chaos seems to continue. Could it be that this fix can't be rolled out automatically to affected machines because they crash during boot - before the Crowdstrike Updater runs?

That update is so tone-deaf and half-assed. There's no apology.

If you go to the website, there's nothing on their front-page. The post on their blog (https://www.crowdstrike.com/blog/statement-on-windows-sensor...) doesn't even link to the solution. There's no link to "Support Portal" anywhere to be seen on their front-page. So, you have to go digging to find the update.

And the "Fix" that they've "Deployed" requires someone to go to Every. Single. Machine. Companies with fleets of 50k machines are on this HN thread - how are they supposed to visit every machine?!?!

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#478
The Windows ecosystem typically deployed in corporate PCs or workstations is often insecure, slow, and poorly implemented, resulting in ongoing issues visible to everyone. Examples include problems with malware, ransomware, and Windows botnets.

In corporate environments, IT staff struggle to contain these issues using antivirus software, firewalls, and proxies. These security measures often slow down PCs significantly, even on recent multi-core systems that should be responsive.

Microsoft is responsible for providing an operating system that is inherently insecure and vulnerable. They have prioritized user lock-in, dark patterns, and ease of use over security.

Apple has done a much better job with macOS in terms of security and performance.

The corporate world is now divided into two categories: 1. Software-savvy companies that run on Linux or BSD variants, occasionally providing macOS to their employees. These include companies like Google, Amazon, Netflix, and many others. 2. Companies that are not software-focused, as it's not their primary business. These organizations are left with Microsoft's offerings, paying for licenses and dealing with slow and insecure software.

The main advantage of Microsoft's products is the Office suite: Excel, Word and Powerpoint but even Word is actually mediocre.

EDIT: improve expression and fix errors:

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#479
post #349

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer startin…

Can anyone explain how CrowdStrike could possibly fix this now? If affected machines are stuck in an endless BSOD cycle, is it even possible to remotely roll out a fix? My understanding is that the machines will never come to the point where a CS update would be automatically installed. Is the only feasible option the official workaround of manually deleting system files after booting into the recovery environment? How could this possibly be done on scale in organizations with tens of thousands of machines?

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#480
I know I have the benefit of hindsight in this regard, but how isn't there redundant checks and tests that would prevent a mishap of this magnitude?

I mean, there should be extensive automated testing using many different platforms and hardware combinations as a prerequisite for any rollout.

I guess this is what we get when everything is opaque, not only the product and the code, but also the processes involved in maintaining and evolving the solution. They would think twice about not investing heavily in testing their deployment pipelines if everyone could inspect their processes.

It might also be the case that they indeed have a thorough production and testing process deployed to support the maintenance of crowdstrike solutions, but we are only left to wonder and to trust whatever their PR will eventually throw at us, since they are a closed company.

Post reply on HN