How do you patch software causing a BSOD?
It seems like a chicken and egg problem.
I ran a team that developed a remote agent, and this was my nightmare scenario.
CrowdStrike Update: Windows Bluescreen and Boot Loops
841–850 of 1001 posts
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#842Read on Mastodon: https://infosec.exchange/@littlealex/112813425122476301 The CEO of Crowdstrike, George Kurtz, was the CTO of McAfee back in 2010 when it sent out a bad update and caused similar issues worldwide. If at first you don't succeed, .... ;-) j/k
fool me once...
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#843Throwaway account... CrowdStrike in this context is a NT kernel loadable module (a .sys file) which does syscall level interception and logs then to a separate process on the machine. It can also STOP syscalls from working if they are trying to connect out to other nodes and accessing files they shouldn't be (using some drunk ass heuristics). What happened here was they pushed a new kernel driver out to every client…
> Most of our nodes are boot looping with blue screens which in the cloud is not something you can just hit F8 and remove the driver.
It took a bit to figure out with some customers, but we provide optional VNC access to instances at OCI, and with VNC the trick seems to be to hit esc and then F8, at the right stage in the boot process. Timing seems to be the devil in the details there, though. Getting that timing right is frustrating. People seem to be developing a knack for it though.
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#844Hopefully now people might wake up to the idea that these tech monopolies are not leading to safe, secure and reliable systems. They will wonder how a third party component could cause such breakage. I expect many will be calling for regulation.
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#845I don’t really understand why AV updates aren’t tested before being pushed out to critical systems and I don’t understand why every system would run the same AV.
But also I don’t understand why this corporate garbageware is still a thing in 2024 when it adds so little value.
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#846Some Canonical guy I think many years ago mentioned this as their sales strategy a few year ago after a particularly nasty Windows outage: We don't ask customers to switch all systems from Windows to Ubuntu, but to consider moving maybe a third to Ubuntu so they won't sit completely helpless next time Windows fail spectacularly. While I see more and more Ubuntu systems, and recently have even spotted Landscape in the…
Many years ago an Ubuntu tech sales guy demoed their (openstack?) Self hosted cloud offering, his laptop was running windows..
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#847Yet their stock tanked only a couple of dollars. They (and their customers) should face some rather unpleasant lawsuits. If you let others own your systems, you should not be allowed to provide critical infrastructure.
I mean, seriously: You can cause a worldwide outage of gargantuan proportions, affecting actual human lives and untold points off GDP in several countries ...
... and the market gives you the equivalent of a wrist slap? No lawsuits?
What is it down to? Anonymous hit markets? Where is justice going to be served here?
PS. Most outlets are reporting "a fix has been issued" - as in "Whoopsy. No biggie" ...
... I mean, who's going to make affected (still alive!) people whole?
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#848why the fuck is our critical infrastructure running on WINDOWS. Fuck the sad state of IT. CIOs and CTOs across the board need to be fired and held accountable for their shitty decisions in these industries.
yes CRWD is a shitty company but seems they are a "necessity" by some stupid audit/regulatory board that oversees these industries. But at the end of the day, these CIOs/CTOs are completely fucking clueless as to the exact functions this software does on a regular basis. A few minions might raise an issue but they stupidly ignore them because "rEgUlAtOrY aUdIt rEqUiReS iT!1!"
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#849Some Canonical guy I think many years ago mentioned this as their sales strategy a few year ago after a particularly nasty Windows outage: We don't ask customers to switch all systems from Windows to Ubuntu, but to consider moving maybe a third to Ubuntu so they won't sit completely helpless next time Windows fail spectacularly. While I see more and more Ubuntu systems, and recently have even spotted Landscape in the…
Linux and open source also have the potential to be far more modular than Windows is. At the moment we have airport display boards running a full windows stack including anti-virus/spyware/audit etc, just to display a table ... madness
Re: CrowdStrike Update: Windows Bluescreen and Boot Loops
#850(FORCE) Pusheedd to Prod on FRIDAYY -- Burneeeddd by its Sins