Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints

techcommunity.microsoft.com

Post reply on HN
Print view View original

51–60 of 66 posts

Re: New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints

#51

Post by windexh8er »

rty32 wrote: post #18
windexh8er wrote: post #13

Earlier quoted context omitted.

> Microsoft has higher quality assurance concerns... No, they don't. This is the same company that has turned the Windows OS into an advertisement platform within the OS [0]. A company that puts buggy telemetry collection over their end users [1]. And a platform that is known to spy on its end users [2]. So, no - Microsoft really doesn't care about its end users with "higher quality assurance concerns". They care abo…

Which is completely irrelevant and does not negate parent comment's point. A real argument would be very informative, but yours just ruined that and is not much different from trolling.

Did you go read the comments in the link above for the Microsoft tool? Because your comment indicates you didn't. I stand by what I said and it does showcase the level of quality Microsoft puts into their products today.

Re: New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints

#52

Post by HPsquared »

capitainenemo wrote: post #36
jorvi wrote: post #35

Earlier quoted context omitted.

One thing I’ve never understood about “kernel never breaks user space”.. doesn’t that completely atrophy the kernel, preventing it from ever having big rewrites or architectural changes? What if an initial implantation was terrible, and there are 100x performance improvements to be had by doing a breaking change?

Implement a new API for the better route, isolate the terrible code as much as possible, notify the users, deprecate it, and remove it or move it to a userspace shim after enough years had past and almost everyone was off it?

That must be a pretty well-worn path by now.

Re: New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints

#53

Post by SushiHippie »

augusto-moura wrote: post #50
hedora wrote: post #44

Earlier quoted context omitted.

Yep; was at a restaurant yesterday, and it sounds like they got hit with the CloudStrike Linux outage a month or so ago. They had no idea the two were probably the same vendor.

Sorry for the ignorance, but what is this Crowdstrike Linux outage you mention? Couldn't find any easily accessible news on it

https://news.ycombinator.com/item?id=41018029

Re: New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints

#54

Post by codebolt »

Arwill wrote: post #25
Bognar wrote: post #7

Earlier quoted context omitted.

They could say "third party kernel modules are installed at your own risk" and provide the usual level of business hours support. CrowdStrike fucked up and Microsoft is helping its customers recover from CrowdStrike's fuckup.

If anything, then events like this makes decision makers rethink if they really should run Windows everywhere. Why does a flight schedule display has to run Windows, for example? It might not be their fuckup, but they will lose users too, for sure.

Same thing already happened on Linux, but it failed to make a big enough splash to make any headlines. Putting Windows at fault here is unfair.

https://www.newsbytesapp.com/news/science/before-affecting-w...

Re: New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints

#55

Post by kchr »

Kwpolska wrote: post #21
etskinner wrote: post #2

Really impressive that they got thru an entire develop, build, approval, and documentation process in just about 2 days. Not that any of those steps are extremely hard for this fix, but I'm always impressed when big corporations can move so fast

To be fair, there isn't a whole lot of code there. I wouldn't be surprised if Microsoft had the WinPE generator written already for some other project.

Yeah, WinPE media tools have been around for years. Here is an article from 2021 (although it has been a thing long before then):

https://learn.microsoft.com/en-us/windows-hardware/manufactu...

Still, customizing the toolchain to fit this particular scenario and making sure it works, in two days, is commendable effort.

Re: New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints

#56

Post by bloopernova »

baq wrote: post #34
bloopernova wrote: post #28

Did anyone write a script to remove the file directly from VM disks, rather than booting the OS? Or does crowdstrike somehow prevent that solution?

I imagine having an unencrypted disk in 2024 can be most charitably called 'an oversight', so there's little point in attempting to deal with them. (Remember we're talking about boxes with crowdstrike installed...)

Ahh, right. You'd need bitlocker keys. Although I wonder if the central key server could be queried to obtain each host's key?

Also makes me wonder about a software configuration management system that operated on disks while the virtual hosts were powered down. With windows it feels like that'd be at least very difficult, but Linux could definitely be managed that way. Like an immutable operating system where changes can only come from the central controller, and the OS itself is written with that in mind. Dunno what benefit that might bring, but it's a fun mental excursion.

Re: New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints

#57

Post by selykg »

nerdjon wrote: post #5
etskinner wrote: post #2

Really impressive that they got thru an entire develop, build, approval, and documentation process in just about 2 days. Not that any of those steps are extremely hard for this fix, but I'm always impressed when big corporations can move so fast

I sympathize with the engineers, QA, and everyone involved in getting this out. I have to imagine it was a lot of long hours, and the testing was insane. The last thing I want to do is put this tool out and it somehow messes things up more. But glad it’s out. Hopefully it helps with the remaining machines and with any that are being problematic.

I wrote and maintained a tool that was sent to users when something went wrong.

Man was I always afraid and stressed that the tool meant to help users when they were already having a failure was also having a failure.

Re: New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints

#58

Post by beefnugs »

etskinner wrote: post #2

Really impressive that they got thru an entire develop, build, approval, and documentation process in just about 2 days. Not that any of those steps are extremely hard for this fix, but I'm always impressed when big corporations can move so fast

Actually they may not have a choice, since they have forced people to install their local windows with a Microsoft login, and tying bitlocker to this login, there is probably many situations out there that requires microsoft login supported winPE just to fix this

Re: New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints

#59

Post by concerned_user »

LASR wrote: post #41
concerned_user wrote: post #12

Earlier quoted context omitted.

I also think Microsoft should be responsible, they gave the keys to sign the kernel driver so I expect that driver to at least be subject to regular testing and scrutiny not just when initial release was made.

Signing is meant only to verify the identity of the organization producing the signed artifact. It’s not meant to signify that it’s bug-free.

WHQL signifies it is tested and that driver is WHQL certified.

Re: New Recovery Tool to help with CrowdStrike issue impacting Windows endpoints

#60

Post by concerned_user »

_flux wrote: post #17
Kwpolska wrote: post #14

Earlier quoted context omitted.

The issue was caused by a data file, Microsoft is not involved in signing or testing individual data files.

The actual issue was with the signed code reading the data files that the data file update just brought to surface. But I don't think Microsoft verifies customer code, they might not even have access to it.

You are right Microsoft are not checking the 3rd party code itself they are only running a lot of tests on the compiled code.

There is a recent video now from a former Microsoft employee where he explains that those drivers that get WHQL certification are ran on test machines in stress conditions for some time, or at least that is how it used to be when he worked there.

Since that process is probably quite slow to be able to push update within a couple hours Crowdstrike just bypassed the QA testing by injecting their own data files into the driver.

Post reply on HN