Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CrowdStrike Update: Windows Bluescreen and Boot Loops

old.reddit.com

Post reply on HN
Print view View original

931–940 of 1001 posts

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#931

Post by TeeWEE »

rkagerer wrote: post #929

This is why I don't like fully automatic updates. I prefer having control over the "deploy" button for the ability to time it when I can tolerate downtime. In mission-critical production systems all updates should go through test staging pipelines that my team controls, not a vendor. Broken updates have cause far more havoc than being a few hours or even days late on a so-called critical patch.

Even if you deploy manually all at once: you have the same problem.

A solution is slow rollout. Not a manual deploy button

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#932

Post by raverbashing »

drooopy wrote: post #174

I've picked the perfect day to return from vacation. Being greeted by thousands of users being mad at you and people asking for your head on a plate makes me reconsider my career choice. Here's to 12 hours of task force meetings...

> and people asking for your head on a plate

I'd say I'd given what they want and left to get an icecream at the part

Let then try to fix it themselves

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#933

Post by jjav »

tbatchelli wrote: post #665

This event is predicted in Sydney Dekker’s book “Drift into Failure”, which basically postulates that in order to prevent local failure we setup failure prevention systems that increase the complexity beyond our ability to handle, and introduce systemic failures that are global. It’s a sobering book to read if you ever thought we could make systems fault tolerant.

> if you ever thought we could make systems fault tolerant

The only possible way to fault tolerancy is simplicity and then more simplicity.

Things like crowsdtrike have the opposite approach. Add a lot of fragile complexity attempting to catch problems, but introducing more attack surfaces than they can remove. This will never succeed.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#934

Post by HarryTom »

jmcgough wrote: post #131

Took down our entire emergency department as we were treating a heart attack. 911 down for our state too. Nowhere for people to be diverted to because the other nearby hospitals are down. Hard to imagine how many millions of not billions of dollars this one bad update caused.

[dead]

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#935

Post by usr1106 »

steelframe wrote: post #640

Wow, this hits close to home. Doing a page fault where you can't in the kernel is exactly what I did with my very first patch I submitted after I joined the Microsoft BitLocker team in 2009. I added a check on the driver initialization path and didn't annotate the code as non-paged because frankly I didn't know at the time that the Windows kernel was paged. All my kernel development experience up to that point was wi…

> I didn't know at the time that the Windows kernel was paged.

At uni I had a professor in database systems, who did not like written exams, but mostly did oral exams. Obviously for DBMSes the page buffer is very relevant, so we chatted about virtual memory and paging. So in my explanation I made the difference for kernel space and user space. I am pretty sure I had read that in a book describing VAX/VMS internals. However, the professor claimed that a kernel never does paging for its own memory. I did not argue on that and passed the exam with the best grade. Did not check that book again to verify my claim. I have never done any kernel space development even vaguely close to memory management, so still today I don't know the exact details.

However, what strikes me here: When that exam happened in 1985ish the NT kernel did not exist yet, I'd believe. However, IIRC a significant part of the DEC VMS kernel team went to Microsoft to work on the NT kernel. So the concept of paging (a part of) kernel memory went with them? Whether VMS --> WNT, every letter increased by one is just a coincidence or intentionally the next baby of those developers I have never understood. As Linux has shown us today much bigger systems can be successfully handled without the extra complications for paging kernel memory. Whether it's a good idea I don't know, at least not a necessary one.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#936

Post by usr1106 »

steelframe wrote: post #640

Wow, this hits close to home. Doing a page fault where you can't in the kernel is exactly what I did with my very first patch I submitted after I joined the Microsoft BitLocker team in 2009. I added a check on the driver initialization path and didn't annotate the code as non-paged because frankly I didn't know at the time that the Windows kernel was paged. All my kernel development experience up to that point was wi…

> It blows my mind that a kernel driver with the level of proliferation in industry could make it out the door apparently without even the most basic level of qualification.

Discussed elsewhere it is claimed that the file causing the crash was a data file that has been corrupted in the delivery process. So the development team and their CI have probably tested a good version, but the customer received a bad one.

If that is true to problem is that the driver first uses an unsigned file at all, so all customer machines are continuously at risk for local attacks. And then it does not do any integrity check on the data it contains, which is a big no no for all untrusted data, whether user space or kernel.

Re: CrowdStrike Update: Windows Bluescreen and Boot Loops

#940

Post by kwhitefoot »

Can someone explain to me why such systems need anti-virus in the first place?

Windows has pretty good facilities for locking down the system so that ordinary users, even those with local admin rights, cannot run or install unauthorised code so if nothing can get in why would the system need checking for viruses?

So why do most companies not lock down their machines?

Post reply on HN