Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike Update: Windows Bluescreen and Boot Loops

old.reddit.com

Post reply on HN
Print view View original

941–950 of 1001 posts

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#941

Post by thih9 »

techbrovanguard wrote: post #143

i've seen photos of the bsod from an affected machine, the error code is `PAGE_FAULT_IN_NONPAGED_AREA`. here's some helpful takeaways from this incident: 1) mistakes in kernel-level drivers can and will crash the entire os 2) do not write kernel-level drivers 3) do not write kernel-level drivers 4) do not write kernel-level drivers 5) if you really need a kernel-level driver, do not write it in a memory unsafe langua…

Other takeaways:

- do not put critical infrastructure online

- do not push updates that work around the update schedule

- do not push such updates to all machines at once

- do not skip testing and QA, relevant to the number and kind of the machines affected

Even one of these would have massively improved the situation, even with a kernel-level driver written in an unsafe language.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#942

Post by miffy900 »

cookiengineer wrote: post #918

I'm a little late to the party, but I've uploaded my source codes to GitHub in case anyone needs a more convenient tool to deploy/execute on running machines and/or needs something fast on USB flash drives to run around the office: https://github.com/cookiengineer/fix-crowdstrike-bsod Releases section contains prebuilt binaries, but of course, I always recommend to check the source and then build it yourself.

I'm sure you mean well, but it's not going to be most programmers or devs who will need to apply a fix for this, it'll be sysadmin/network/SREs who'll be doing this and they're not going to download Go to build this random github code repo. Because it affects only Windows systems, it'll be way better writing a bat or powershell script that can non-programmers can read and comprehend before they execute anything in production/live systems.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#943

Post by tomthumb »

Someone on X has shared the kernel stack trace of the crash

The faulting driver in the stack trace was csagent.sys.

Now, Crowdstrike has got two mini filter drivers registered with Microsoft (for signing and allocation of altitude).

1) csagent.sys - Altitude (321410) This altitude falls within the range for Anti-Virus filters. 2) im.sys - Altitude (80680) This altitude falls within the range for access control drivers.

So, it is clear that the driver causing the crash is their AV driver, csagent.sys.

The workaround that CrowdStrike has given is to delete C-00000291*.sys files from the directory: C:\Windows\System32\Drivers\CrowdStrike\

These files being suggested to be deleted are not driver files (.sys files) but probably some kind of virus definition database files.

The reason they name these files with the .sys extension is possibly to leverage Windows System File Checker tool's ability to restore back deleted system files.

This seems to be a workaround and the actual fix might be done in their driver, csagent.sys and the fix will be rolled out later.

Anyone having access a Falcon endpoint might see a change in the timestamp of the driver csagent.sys when the actual fix rolls out.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#944

Post by rldjbpin »

it is humbling (and lowkey reassuring?) to know that not all large players use the absolute cutting edge approaches in their workflow.

it seems and i hope that after all is said and done there is no major life-threatening consequence of this debacle. at the same time, heart goes out to the dev who pushed the troubling code. very easy to point at them or the team's processes, but we need to introspect at our own setup and also recognize that not all of us work in crucial systems like this.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#948

Post by sandworm101 »

steelframe wrote: post #640

Wow, this hits close to home. Doing a page fault where you can't in the kernel is exactly what I did with my very first patch I submitted after I joined the Microsoft BitLocker team in 2009. I added a check on the driver initialization path and didn't annotate the code as non-paged because frankly I didn't know at the time that the Windows kernel was paged. All my kernel development experience up to that point was wi…

So the key test, the test that was not run, was to turn the machine off and on again? Classic windows.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#950

Post by sphar1970 »

teeheelol wrote: post #203

Throwaway account... CrowdStrike in this context is a NT kernel loadable module (a .sys file) which does syscall level interception and logs then to a separate process on the machine. It can also STOP syscalls from working if they are trying to connect out to other nodes and accessing files they shouldn't be (using some drunk ass heuristics). What happened here was they pushed a new kernel driver out to every client…

But it is Windows because the kernel should be able to roll back a bad update, there should NEVER be BSODs.
Post reply on HN