Live data from Hacker News

CrowdStrike Update: Windows Bluescreen and Boot Loops

old.reddit.com

681–690 of 1001 posts

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#682
Why are so many mission critical hardware connected systems connected to the internet at all or getting automatic updates?

This is just basic IT common sense. You only do updates during a planned outage, after doing an easily reversible backup, or you have two redundant systems in rotation and update and test the spare first. Critical systems connected to things like medical equipment should have no internet connectivity, and need no security updates.

I follow all of this in my own home so a bad update doesn’t ruin my work day… how do big companies with professional IT not know this stuff?

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#686

Due to the scale I think it’s reasonable to state that in all likelihood many people have died because of this. Sure it might be hard to attribute single cases but statistically I would expect to see a general increase in probability. I used to work at MS and didn’t like their 2:1 test to dev ratio or their 0:1 ratio either and wish they spent more work on verification and improved processes instead of relying on tes…

I love their forced updates, because if you know what you're doing you can disable them, and if you don't know what you're doing, well you shouldn't be disabling updates to begin with. I think people forget how virus infested and bug addled Windows used to be before they enforced updates. People wouldn't update for years and then bitch how bad Windows was, when obviously the issue wasn't Windows at that point.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#688

Perhaps a dumb question for someone who actually knows how Microsoft stuff works... Why would an anti-malware program be allowed to install a driver automatically ... or ever for that matter? Added: OK, from another post I now know Crowdstrike has some sort of kernel mode that allows this sort of catastrophe on Linux. So I guess there is a bigger question here...

As I understand it was a definition update that caused a crash inside already installed driver.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#689
post #538

Crowdstrike did this to our production linux fleet back on April 19th, and I've been dying to rant about it. The short version was: we're a civic tech lab, so we have a bunch of different production websites made at different times on different infrastructure. We run Crowdstrike provided by our enterprise. Crowdstrike pushed an update on a Friday evening that was incompatible with up-to-date Debian stable. So we patc…

Please tell me you've ended the contract with CrowdStrike after this?

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#690
AWS has posted some instructions for those affected by the issue using EC2.

[AWS Health Dashboard](https://health.aws.amazon.com/health/status)

"First, in some cases, a reboot of the instance may allow for the CrowdStrike Falcon agent to be updated to a previously healthy version, resolving the issue.

Second, the following steps can be followed to delete the CrowdStrike Falcon agent file on the affected instance:

1. Create a snapshot of the EBS root volume of the affected instance

2. Create a new EBS volume from the snapshot in the same Availability Zone

3. Launch a new instance in that Availability Zone using a different version of Windows

4. Attach the EBS volume from step (2) to the new instance as a data volume

5. Navigate to the \windows\system32\drivers\CrowdStrike\ folder on the attached volume and delete "C-00000291*.sys"

6. Detach the EBS volume from the new instance

7. Create a snapshot of the detached EBS volume

8. Create an AMI from the snapshot by selecting the same volume type as the affected instance

9. Call replace root volume on the original EC2 Instance specifying the AMI just created"

Post reply on HN