Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike Update: Windows Bluescreen and Boot Loops

old.reddit.com

Post reply on HN
Print view View original

281–290 of 1001 posts

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#281

Post by Frieren »

openrisk wrote: post #196

The details (the particular companies / systems etc) of this global incident don't really matter. When the entire society and economy are being digitized AND that digitisation is controlled and passes through a handful of choke points its an invitation to major disaster. It is risk management 101, never put all your digital eggs in one (or even a few) baskets. The love affair with oligopoly, cornered markets and powe…

Efficiency goes against resilience.

Can you imagine having just one road connecting two big cities to cut costs? No alternative roads, nor big nor small.

That will be really cheap to maintain, and you can charge as much as you want in tolls as there are no alternatives. And you can add ads all over the road as people has to watch them to move from one city to the other.

And if the road breaks, the goverment needs to pay for the cost as they cannot allow the cities to go unconnected.

We live in the middle-ages of technology.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#282

Post by baxtr »

openrisk wrote: post #196

The details (the particular companies / systems etc) of this global incident don't really matter. When the entire society and economy are being digitized AND that digitisation is controlled and passes through a handful of choke points its an invitation to major disaster. It is risk management 101, never put all your digital eggs in one (or even a few) baskets. The love affair with oligopoly, cornered markets and powe…

It is also risk management 101 that managing (i.e. avoiding or insuring) risks doesn't come for free.

The cost and benefits of managing risks need to be balanced.

So I am not convinced that there need to be "at least ten alternatives" to be fail safe as society.

Where I agree with you is that these decisions should be deliberate and done after a cost / benefit analysis.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#284

Post by vesinisa »

WatchDog wrote: post #257

This was apparently caused by a faulty "channel file"[0], which is presumably some kind of configuration database that the software uses to identify malware. So there wasn't any new kernel driver deployed, the existing kernel driver just doesn't fail gracefully. [0]: https://x.com/brody_n77/status/1814185935476863321

Why on earth don't they have staged rollouts for updates?

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#285

Post by commercialnix »

All my customers endpoints are on Linux based endpoints. Because our users' Windows apps run in vdi with disposable instances based off snapshots and highly restrictive networking on the Linux endpoints, none of our users are effected.

Running Windows on bare-metal was always obviously very stupid. The consequences of such stupidity are just being felt now.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#286

Post by teeheelol »

openrisk wrote: post #196

The details (the particular companies / systems etc) of this global incident don't really matter. When the entire society and economy are being digitized AND that digitisation is controlled and passes through a handful of choke points its an invitation to major disaster. It is risk management 101, never put all your digital eggs in one (or even a few) baskets. The love affair with oligopoly, cornered markets and powe…

That's not really the problem here.

The problem is allowing a single vendor, with a reputation of fucking up over and over again, to push code into your production systems at will with no testing on your part.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#287

Post by CaliforniaKarl »

shubhamjain wrote: post #218

Naive question, if it’s a blue screen of death with a boot loop, how are they going to restore things? Don’t tell me the answer is going to every system manually.

Lots of systems (not all) are able to reboot, and have CrowdStrike download the fix before the bad code is able to crash things. But otherwise, yes, you have to go to systems manually.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#288

Post by aboringusername »

openrisk wrote: post #196

The details (the particular companies / systems etc) of this global incident don't really matter. When the entire society and economy are being digitized AND that digitisation is controlled and passes through a handful of choke points its an invitation to major disaster. It is risk management 101, never put all your digital eggs in one (or even a few) baskets. The love affair with oligopoly, cornered markets and powe…

> there should be at least ten alternatives in any diversified set of critical infrastructure service providers, all of them instantly replaceable / forced to provide interoperability...

And does anyone actually know how to actually implement this, at the scale required (dealing with billions of transactions daily) in a way that would resolve the problems we are seeing?

It very much seems like a data access problem; places can't access/modify data. The physical disks themselves are most likely fine, but the 'interfaces' are having troubles (assuming that the data isn't stored on the devices having the issue).

But in any case how do you design a system that, if the 'main' interface is troubled, you can switch over, instantly, seemlessly, duplicating access controls, permissions, data validation, logic etc.

There is a reason everything is centralised because it makes no financial sense to duplicate for an extremely unlikely and rare chance. The world is random and these things will happen, but a global outage on this type of scale is not a daily occurance.

We'll look back in a few years and think "those were a crazy few hours" and move on...

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#289

Post by ogurechny »

shubhamjain wrote: post #218

Naive question, if it’s a blue screen of death with a boot loop, how are they going to restore things? Don’t tell me the answer is going to every system manually.

Well, it seems that Windows is not yet accessible remotely when it crashes.

If system administrator had too much free time, and configured every system to probe network on booting, and there is no encryption, it is possible to boot from a minimal Linux image with a script that automatically renames the driver and restarts.

The corporate version of the same approach uses Intel AMT (or however else it is called), but it is only available on licensed hardware from big suppliers.

Otherwise, you can distribute flash drives with the same auto-executing fix to everyone who is able to enter firmware setup, and boot from USB. If it's not available for security reasons, more manual work is required.

But what happens next? If Crowdstrike handled all the security measures, and there was no additional firewall rules, address checks, and so on, your network is now as open as it can be. I suppose certain groups have been celebrating, and uploading gigabytes of data from networks whose detection systems became severed.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#290

Post by skerit »

nmcveity wrote: post #113

This gem from the ABC news coverage has my mind 100% boggled: "711 has been affected by the outage … went in to buy a sandwich and a coffee and they couldn’t even open the till. People who had filled up their cars were getting stuck in the shop because they couldn’t pay." Can't even take CASH payment without the computer, what a world!

I'm seeing several reports of things like being unable to buy tickets for the train on-line in Belgium.

They use Windows as a part of their server infrastructure?

Post reply on HN