Live data from Hacker News

Microsoft's global sprawl under fire from regulators after Windows outage

washingtonpost.com

71–80 of 104 posts

Re: Microsoft's global sprawl under fire from regulators after Windows outage

#71
post #54
post #6

It seems baffling that Microsoft is getting heat for this. They didn't cause the issue, a third party vendor's software did. Even if you were trying to make an argument of "if we had more diversity it wouldn't be as bad", shouldn't you be focusing on the EDR vendors rather than the OS vendor?

There is a frustrating amount of nuance being lost in this discussion, and as usual it's devolving into tribalism. However, I'll say that, while this clearly is not Microsoft's fault, the realization of just how much critical infrastructure is running on Windows -- let alone Windows that's connected to the internet and has automatic updates enabled -- was sobering. Are kernel mode drives maybe a bad idea? Yes! Should…

But as you've outlined multiple times, the flaws of the OS are not the problem here. Changing OS, the same poor decisions can and have been made. Windows can be locked down or configured to a very stable level, and Linux can be configured to a shitshow.

I don't like Windows, but embedded edition has been the most common operating system you encounter in the physical world for decades, and broadly it works so well you don't know. I don't see a good argument that something on the Windows end needs fixing here.

Re: Microsoft's global sprawl under fire from regulators after Windows outage

#72

Do some of these news agencies own Crowdstrike stock? How can you blame Windows here? For simply allowing an application that can crash the OS? I truly don't understand the angle here, besides ignorance.

From the point of view of a lay person Microsoft was caught asleep at the wheel. It allowed high levels of privileged access to their customers' systems. Anti-malware software ought to be monitored and scrutinised by Microsoft's own security teams no matter that it was the customers who wanted to/were told to use CrowdStrike. But I guess they spend too much on silly fidget-widgets like OpenAI to have a budget to watc…

If I design and sell a car and you electrocute yourself replacing a head unit cutting wires - despite warnings from me saying "I don't recommend replacing the head unit, it could be dangerous. I've provided you one with basic functionality" - can you reasonably argue that I should have made it harder to replace head units, or been out auditing third party head units on the market?

Your argument is anti software freedom. My Windows computer (or any other OS I choose to put on) should be able to run whatever code I instruct it to. The flip side of that coin is that the consequences are mine from doing so.

Re: Microsoft's global sprawl under fire from regulators after Windows outage

#73
post #54
post #6

It seems baffling that Microsoft is getting heat for this. They didn't cause the issue, a third party vendor's software did. Even if you were trying to make an argument of "if we had more diversity it wouldn't be as bad", shouldn't you be focusing on the EDR vendors rather than the OS vendor?

There is a frustrating amount of nuance being lost in this discussion, and as usual it's devolving into tribalism. However, I'll say that, while this clearly is not Microsoft's fault, the realization of just how much critical infrastructure is running on Windows -- let alone Windows that's connected to the internet and has automatic updates enabled -- was sobering. Are kernel mode drives maybe a bad idea? Yes! Should…

Totally agree here. There should be a mechanism to go back to ‘last known good’ regardless of kernel level issues. Innovations like Fedora Silverblue with ostree and greenboot tech should be adopted by Windows.

Re: Microsoft's global sprawl under fire from regulators after Windows outage

#74
post #60
post #54

Earlier quoted context omitted.

There is a frustrating amount of nuance being lost in this discussion, and as usual it's devolving into tribalism. However, I'll say that, while this clearly is not Microsoft's fault, the realization of just how much critical infrastructure is running on Windows -- let alone Windows that's connected to the internet and has automatic updates enabled -- was sobering. Are kernel mode drives maybe a bad idea? Yes! Should…

>There is a frustrating amount of nuance being lost in this discussion, and as usual it's devolving into tribalism. It's ironic that you're saying this given the points you're making below. Let's go through them: >Are kernel mode drives maybe a bad idea? Yes! You can't have a EDR product that isn't kernel mode. Otherwise it's trivial for malware to evade (eg. by being kernel mode themselves). >Should Windows be able…

> It's ironic that you're saying this given the points you're making below.

The nuance I was referring to was with regards to the actual facts of the situation, such as who is responsible, and that many, many people are just using it as an excuse to dunk on Windows. As I said, Windows/Microsoft are not at fault in this precise situation.

However, is it possible that I can state that a thing is bad without it being "tribalism?"

I'm invoking tribalism and setting myself apart from it in an attempt to make it very clear that my criticism of Windows is not simply tribalism. Stating that Boeing airplanes are prone to critical faults due to bad engineering is a fact, not tribalism; that is still true even if I personally dislike Boeing airplanes/the company. Perhaps I'm even critical of them precisely because of the bad engineering I've observed! Weird how that works.

The same can be said for Windows.

> Getting companies to test updates is already like pulling teeth. Besides, crowdstrike said the update they pushed was "designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks". Is this something you really want to sit on for testing, which might take weeks or months?

gestures broadly I mean...given the current situation, obviously I'm going to answer with an emphatic "yes!" You know we have these things called computers, right? They're really good at automating stuff. Like testing.

I know that "getting companies to test updates is like pulling teeth," but that doesn't mean it shouldn't be done. Companies do all sorts of stupid and negligent bullshit, are happy to spend money in the name of shifting blame, but are cheap as hell with regards to actually avoiding problems. That's not a good thing, and it should change. Is that really such a controversial statement? Apologies for potentially engaging in strawmanning, but in the even that your response is something along the lines of "they should test, but it's not realistic to expect that," yeah, I agree, but perhaps this precise event is the kick in the pants those who are against testing need to stop being cheap morons. And in case you're not clean on who I'm referring to: the decision makers at the top, not the engineers caring out their irresponsible and negligent agendas.

I was in the ER quite literally the day before this hit with a slash to my popliteal artery (long story, freak accident), and I shudder to think how it would have gone a day later -- I honestly could have bled out and died. The fact that so many places running absolutely critical infrastructure aren't routinely testing every change they push out to their devices is insane. Utterly, bat shit insane.

> What specific security issues do you think windows/NT kernel has?

Sorry, not taking the bait on this one. Windows is a piece of shit, and it's absolutely self-evident to anyone even kind of exposed to the alternatives. Expanding on this to you is a waste of time, as either you've never used Windows before (highly unlikely) or you're unwilling to see what I'm talking about for whatever reason.

> Moreover, how is windows being a "shitshow" relevant to the question of resiliency or dependence?

Do I really have to explain to you why a shitshow of an OS isn't resilient?

I've tried to make it extremely clear that 1) I don't blame Windows or Microsoft in this incident, but 2) this incident revealed just how much critical infrastructure relies on an OS that has no business being used as such. That's not "don't let a disaster go to waste," it's one disaster revealing a situation that is ripe for many, many more. I'm not "anti-Windows," I'm "anti-Windows-as-a-server" and "anti-horrible-system-administration-practices."

Re: Microsoft's global sprawl under fire from regulators after Windows outage

#75
post #70

Earlier quoted context omitted.

>Only windows computers were affected Because only Crowdstrike's Windows release was broken by them and they didn't fuck it up on the other OS. How is this Window's fault? It's not like that tool was binary cross platform compatible for all operating systems, like Electron VS Code, in order to put the blame on the OS. It's basically a complete different tool tailored to each OS kernel, under the same brand name. > if…

> Because only Crowdstrike's Windows release was broken by them and they didn't fuck it up on the other OS. How is this Window's fault? First of all, whether it is actually Window's fault or not isn't the point. What matters is the perception by the general public and policy makers. And maybe it could have just as easily happened on another OS, but the reason why is somewhat technical, so people who have an agenda ca…

>> Because only Crowdstrike's Windows release was broken by them and they didn't fuck it up on the other OS. How is this Window's fault?

>First of all, whether it is actually Window's fault or not isn't the point. What matters is the perception by the general public and policy makers

Way to move the goalposts from "How is this Window's fault" to "the perception by the general public and policy makers". I don't think anyone is disputing that some people are blaming Microsoft for this. In fact the whole impetus for this comment chain is me pointing out how the public perception of microsoft being at fault doesn't match up with logic.

Re: Microsoft's global sprawl under fire from regulators after Windows outage

#76
post #74
post #60

Earlier quoted context omitted.

>There is a frustrating amount of nuance being lost in this discussion, and as usual it's devolving into tribalism. It's ironic that you're saying this given the points you're making below. Let's go through them: >Are kernel mode drives maybe a bad idea? Yes! You can't have a EDR product that isn't kernel mode. Otherwise it's trivial for malware to evade (eg. by being kernel mode themselves). >Should Windows be able…

> It's ironic that you're saying this given the points you're making below. The nuance I was referring to was with regards to the actual facts of the situation, such as who is responsible, and that many, many people are just using it as an excuse to dunk on Windows. As I said, Windows/Microsoft are not at fault in this precise situation. However, is it possible that I can state that a thing is bad without it being "t…

>The nuance I was referring to was with regards to the actual facts of the situation, such as who is responsible, and that many, many people are just using it as an excuse to dunk on Windows.

Right, and I'm pointing out the irony one level down, with some of your takes (ie. not the facts of the situation, but the suggestions that you're making).

>gestures broadly I mean...given the current situation, obviously I'm going to answer with an emphatic "yes!" You know we have these things called computers, right? They're really good at automating stuff. Like testing.

I don't think anyone thinks testing wouldn't have prevented this disaster, nor that testing is bad. The question is whether holding back updates is actually better overall in practice. Remember the Equifax hack? Turned out it was caused by them using a vulnerable version of Apache Struts, which they didn't update for months/years. Now, should they also theoretically have been doing engineering best practices and having a testing pipeline that would allow them to update library versions with minimal fuss? Yes, but in practice that's not something that can be done. The same applies to EDR updates. Should end users' IT departments have test suites so that they can test and release updates within hours of them being released? Yes. Is that a realistic option that actually exists? No.

>> Moreover, how is windows being a "shitshow" relevant to the question of resiliency or dependence?

>Sorry, not taking the bait on this one. Windows is a piece of shit, and it's absolutely self-evident to anyone even kind of exposed to the alternatives. Expanding on this to you is a waste of time, as either you've never used Windows before (highly unlikely) or you're unwilling to see what I'm talking about for whatever reason.

Clearly you don't have a context window exceeding one sentence, because the two sentences immediately following is critical to the understanding of that sentence. If you read those, you'd even see listed out common reasons why people think windows is bad.

>I've tried to make it extremely clear that 1) I don't blame Windows or Microsoft in this incident, but 2) this incident revealed just how much critical infrastructure relies on an OS that has no business being used as such. That's not "don't let a disaster go to waste," it's one disaster revealing a situation that is ripe for many, many more. I'm not "anti-Windows," I'm "anti-Windows-as-a-server" and "anti-horrible-system-administration-practices."

Sounds like you're already convinced that windows is bad, and the only new thing you got out of this is that a lot of important systems run on windows?

Re: Microsoft's global sprawl under fire from regulators after Windows outage

#77
post #54

Earlier quoted context omitted.

There is a frustrating amount of nuance being lost in this discussion, and as usual it's devolving into tribalism. However, I'll say that, while this clearly is not Microsoft's fault, the realization of just how much critical infrastructure is running on Windows -- let alone Windows that's connected to the internet and has automatic updates enabled -- was sobering. Are kernel mode drives maybe a bad idea? Yes! Should…

Totally agree here. There should be a mechanism to go back to ‘last known good’ regardless of kernel level issues. Innovations like Fedora Silverblue with ostree and greenboot tech should be adopted by Windows.

>There should be a mechanism to go back to ‘last known good’ regardless of kernel level issues. Innovations like Fedora Silverblue with ostree and greenboot tech should be adopted by Windows.

Can you explain how they work? AFAIK the issue is that they pushed a bad config file, and that's the thing that caused the crash, not a new driver. Are those systems going to roll back every file ever to try to recover themselves?

Re: Microsoft's global sprawl under fire from regulators after Windows outage

#78

Earlier quoted context omitted.

Ironically I think Microsoft losing windows won't hurt them that much, it will however benefit consumers for sure.

I don’t think Windows would survive for very long without other Microsoft businesses subsidizing their revenue. It might go on for a few years, but it would basically move up Linux and OSX quickly to corporate desktop status, while Windev goes bankrupt as a non-viable business.

The monetization model would indeed be complicated. But, as an off the cuff estimate, there are several billion Windows installations in the world. A small fee (say $2/year on average) would easily cover an excellent engineering operation to maintain Windows. And appropriate regulation could force everyone to use Windows or a similarly independent OS, and the whole industry could survive this.

Re: Microsoft's global sprawl under fire from regulators after Windows outage

#79
post #6

It seems baffling that Microsoft is getting heat for this. They didn't cause the issue, a third party vendor's software did. Even if you were trying to make an argument of "if we had more diversity it wouldn't be as bad", shouldn't you be focusing on the EDR vendors rather than the OS vendor?

I'm curious what people think, but while obviously CrowdStrike caused the breakage, does the Operating System not have some responsibility in not allowing such outages to happen? Especially if it's an enterprise product? Ideas: 1. Microsoft themselves could potentially enforce a gradual rollout on updates (did the update go through windows updates?) 2. Have better automatic recovery options, could windows have detect…

> does the Operating System not have some responsibility in not allowing such outages to happen?

No. External drivers are acting as part of the operating system, their responsibility is to follow the rules for kernel drivers. Don't use-after-free or dereference otherwise bad memory (as appears to be the case here[1]), make sure you only access pageable memory at the correct IRQL, etc etc.

The kernel's job is to provide services and to make sure the other components are running smoothly. The problem is, by the time that something bad like a bad dereference has occurred, other issues may start to arise. And unloading a driver or something may cause data loss and may not actually fix the underlying problem (especially if you pin the error on the wrong driver[2]).

If third party software does not follow the contract, there's... really not much they can or really should do. In user mode, an access violation is given to the program when you access bad memory. This usually results in a process crash, which while annoying, may be fine. User mode programs can't[3] bring down the operating system.

In kernel mode, there's no way for the OS to know that you're not going to start overwriting the disk accidentally so they made what they believe to be the safest choice--stop[4].

In any case, there's really no way for 1 to happen (since driver updates can be done externally to Microsoft), 2 is nebulous, and 3 is potentially dangerous.

---

[1]: https://learn.microsoft.com/en-us/windows-hardware/drivers/d...

[2]: For example, in the case of stack corruption.

[3]: Technically they can in a couple of limited cases (but this really isn't the point). The first being killing CSRSS or another process with the "critical" kernel flag, and another by using NtShutdownSystem from the NT API.

[4]: Other operating systems have taken a different philosophy. Notably Linux can be configured to allow the machine to run after a driver or other external event causes a kernel oops.

Re: Microsoft's global sprawl under fire from regulators after Windows outage

#80
post #6

It seems baffling that Microsoft is getting heat for this. They didn't cause the issue, a third party vendor's software did. Even if you were trying to make an argument of "if we had more diversity it wouldn't be as bad", shouldn't you be focusing on the EDR vendors rather than the OS vendor?

I'm curious what people think, but while obviously CrowdStrike caused the breakage, does the Operating System not have some responsibility in not allowing such outages to happen? Especially if it's an enterprise product? Ideas: 1. Microsoft themselves could potentially enforce a gradual rollout on updates (did the update go through windows updates?) 2. Have better automatic recovery options, could windows have detect…

> does the Operating System not have some responsibility in not allowing such outages to happen?

No. The OS is supposed to guarantee that userspace programs can't crash the system like this, but CrowdStrike is an invasive kernelspace driver, not a userspace program.

Post reply on HN