CrowdStrike Update: Windows Bluescreen and Boot Loops

Lack of gradual, health mediated rollout is absolutely the core issue here. False positive signatures, crash inducing blocks, etc will always slip through testing at some % no matter how good testing is. The necessary defense in depth here is to roll out ALL changes (binaries, policies, etc) in a staggered fashion with some kind of health checks in between (did > 10% of endpoints the change went to go down and stay down right after the change was pushed?).

Crowdstrike bit my company with a false positive that severely broke the entire production fleet because they pushed the change everywhere all at once instead of staggering it out. We pushed them hard in the RCA to implement staggered deployments of their changes. They sent back a 50 page document explaining why they couldn't which basically came down to "that would slow down blocks of true positives" - which is technically true but from followup conversations quite clear that is was not the real reason. The real reason is that they weren't ready to invest the engineering effort into doing this.

You can stagger changes out within a reasonable timeframe - the blocks already take hours/days/weeks to come up with, taking an extra hour or two to trickle the change out gradually with some basic sanity checks between staggers is a tradeoff everyone would embrace in order to avoid the disaster we're living through today.

Need a reset on their balance point of security:uptime.

This is a good example of why you don't want ring0 level access for clients. Or just, you don't want client-based solutions. The provider just becomes another threat vector.

> So we need to hold regulatory bodies accountable as well - when they frame regulation such that organisations are cornered into this they get to be part of the culpability here too.

Replacing common-law liability with prescriptive regulation is one of the main drivers of this problem today. Instead of holding people accountable for the actual consequences of their decisions, we increasingly attempt to preempt their decisions, which is the very thing that incentivizes cargo-cult "checkbox compliance".

It motivates people who otherwise have skin in the game and immediate situational awareness to outsource their responsibility to systems of generalized rules, which by definition are incapable of dealing effectively with outliers.

It's VERY easy to blame CrowdStrike and companies like them as they are the one LOBBYING for those checkboxes. Both zscaler and Crowdstrike spent 500K last year lobbying.

its strange how people who work in professions that are considered crucial infrastructure are held to such a high standard but there's always some tech problem that cripples them the hardest