CrowdStrike Update: Windows Bluescreen and Boot Loops • Hacker News Board

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#481

Post by jojobas » Fri, Jul 19, 2024, 11:42 AM UTC

zmmmmm wrote: post #349

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer startin…

The question is, how did they manage to not crash everything for so long without a staged/rolling update deployment strategy?

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#482

Post by krembo » Fri, Jul 19, 2024, 11:42 AM UTC

kidbomb wrote: post #474

Lessons learned from this: - CS: Have a staging (production-like) environment for proper validation. It looks like CS has one of these bu they have just skipped it - IT Admins: Have controlled roll-outs, instead of doing everything in a single swoop. - CS: Fuzz test your configuration Anything I have missed?

Don't. Deploy. On. Fridays.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#483

Post by jojobas » Fri, Jul 19, 2024, 11:43 AM UTC

DebtDeflation wrote: post #420

At one point overnight airlines were calling for an "international ground stop for all flights globally". Planes in the air were unable to get clearance to land or divert. I don't believe such a thing has ever happened before except in the immediate aftermath of 9/11.

Unbelievable these systems run on Windows.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#484

Post by fnord77 » Fri, Jul 19, 2024, 11:43 AM UTC

I take it patching remote machines is going to be difficult or impossible?

I haven't used windows in years, but from what I read you need to be in safe mode to delete a crowdstrike file in a system directory, but you need some 48 char key to get into safe mode now if it is locked down?

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#485

Post by krembo » Fri, Jul 19, 2024, 11:44 AM UTC

avidphantasm wrote: post #442

How long before companies start consciously de-risking by replacing general-purpose systems like Windows with newer systems with smaller attack surfaces? Why does an airline need to use Windows at all for operations? From what I’ve seen, their backend systems are still running on mainframes. The terminals are accessed on PCs running Windows, but those could trivially be replaced with iPadOS devices that are more lock…

“Nobody ever got fired for Buying IBM”

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#486

Post by red_admiral » Fri, Jul 19, 2024, 11:46 AM UTC

I can't wait for rachelbythebay's comments on this.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#487

Post by gz5 » Fri, Jul 19, 2024, 11:47 AM UTC

Seems CS themselves may have been hacked? For example, seems unlikely that both:

1. CS normally pushes global updates to entire user base simultaneously?

2. This made it through their testing. Not only 'just' QA but likely CS employees internally run a version or two ahead of their customer base?

Just speculation - folks who know either answer can validate or debunk.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#488

Post by Kye » Fri, Jul 19, 2024, 11:48 AM UTC

Crowdstrike seems like the kind of thing that's sold to CEOs at conferences, forced on IT against objections, and the subject of a lot of discussion at Defcon.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#489

Post by EvanAnderson » Fri, Jul 19, 2024, 11:49 AM UTC

I wonder what Crowdstrike's opsec is like re: malicious actors gaining control of their automated update servers. This incident certainly highlights the power of that type of attack, even if this one just ends up being typical human incompetence-based.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#490

Post by ExoticPearTree » Fri, Jul 19, 2024, 11:49 AM UTC

zmmmmm wrote: post #349

So CrowdStrike is deployed as third party software into the critical path of mission critical systems and then left to update itself. It's easy to blame CrowdStrike but that seems too easy on both the orgs that do this but also the upstream forces that compel them to do it. My org which does mission critical healthcare just deployed ZScaler on every computer which is now in the critical path of every computer startin…

> All over the place I'm seeing checkbox compliance being prioritized above actual real risks from how the compliance is implemented.

Because if everyone is doing their job and checks their box, they're not gonna get fired. Might be out of a job because the company goes under, but hey, it was no one's fault, they just did their job.