Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No More Blue Fridays

brendangregg.com

Post reply on HN
Print view View original

211–220 of 282 posts

Re: No More Blue Fridays

#211

Post by hello_moto »

whynotminot wrote: post #204
hello_moto wrote: post #200

Earlier quoted context omitted.

>if you follow industry best practices for software deployment, this is already something that should be considered. This is a relatively solved problem -- this is not new. You have to ask the customer if they're okay with that citing "our software might failed and brick your machine". I'd like to see any Sales and Marketing folks say that ;) > I think that question is being asked right now by every company using Cro…

> You have to ask the customer if they're okay with that citing "our software might failed and brick your machine". I think you’re still missing the point of Canary deployments. The question your sales team should ask is “would you like a 5% chance of a bug harming your system, or a 100% chance?” > It's like: previously Cybersecurity vendors are shy to ask customers to setup Canary systems because that's just "one-mo…

If you refer Canary deployment as the vendor's internal deployment? I definitely agree.

What I find it hard is those in Software that suggested to roll it to a few customers first because this isn't cloud deployment doing A/B test when it comes to Virus Definition.

Customers must know what's going on when it comes to virus definition and the implication of them whether they're being part of the rollout group or not.

Re: No More Blue Fridays

#212

Post by emn13 »

kayo_20211030 wrote: post #15

This isn't right. If I need a system to run with a piece of code, then it shouldn't run at all if that piece of code is broken. Ignoring the failure is perverse. Let's say that the driver code ensures that some medical machine has safety locks (safeguards) in place to make sure that piece of equipment won't fry you to a crisp; I'd prefer that the whole thing not run at all rather than blithely operate with the safegu…

The system clearly already behaves that way (i.e. ignores failure) - after all, the fix was to simply delete the offending file. If that's an option, then loader can do that too. It can and perhaps even is smarter, such as "fallback onto previous version".

Furthermore, the reaction to a malformed state need not be "ignore". It could disable restricted user login; or turn off the screen.

If the worry is that this is viable to abuse by malware, well, if the malware can already rewrite the on-disk files for the AV, I wonder whether it's really a good idea to trust the system itself to be able to deal with that. It'd probably be safer to just report that up the security foodchain, and potentially let some external system take measures such as disable or restrict network access. Better yet, such measures don't even require the same capabilities to intervene in the system, merely to observe - which makes the AV system less likely to serve as a malware vector itself or to cause bugs like this.

Re: No More Blue Fridays

#213

Post by queuebert »

recursive wrote: post #191
queuebert wrote: post #167

Earlier quoted context omitted.

I believe the term you are looking for is "rent seeking". Other than visual changes, what new functionality does Windows 11 actually have that Windows XP didn't have? (I'm being generous with XP, because actually 95 was already mostly internet ready.) Yet how many times have many of us paid for a Windows license on a new computer or because the old version stopped getting updates?

> what new functionality does Windows 11 actually have that Windows XP didn't have? ( Off the top of my head, built-in bluetooth support, an OS-level volume mixer, and more support for a wider variety of class-compliant devices. I'm sure there are a lot more, and if you actually care about the answer, I don't think it would be hard to find.

All of this could've been added to XP, right?

Re: No More Blue Fridays

#214

Post by recursive »

queuebert wrote: post #213
recursive wrote: post #191

Earlier quoted context omitted.

> what new functionality does Windows 11 actually have that Windows XP didn't have? ( Off the top of my head, built-in bluetooth support, an OS-level volume mixer, and more support for a wider variety of class-compliant devices. I'm sure there are a lot more, and if you actually care about the answer, I don't think it would be hard to find.

All of this could've been added to XP, right?

I don't know.

If it could, Then XP would just be Windows 11. What's the objection here.

Re: No More Blue Fridays

#215

Post by wolrah »

queuebert wrote: post #167
kevincox wrote: post #88

Earlier quoted context omitted.

They aren't over windows. They continue to be incredibly interested in and actively developing how much money they can suck from their users. Especially via various forms of ads. But yeah, kernel features are few and far between.

I believe the term you are looking for is "rent seeking". Other than visual changes, what new functionality does Windows 11 actually have that Windows XP didn't have? (I'm being generous with XP, because actually 95 was already mostly internet ready.) Yet how many times have many of us paid for a Windows license on a new computer or because the old version stopped getting updates?

> Other than visual changes, what new functionality does Windows 11 actually have that Windows XP didn't have?

XP->Vista alone brought a bunch of huge changes that massively improved security (UAC), capability (64 bit desktops), and future-proofing (UEFI) among many many other things.

Some helpful Wikipedia editors have answered this question in excessive detail, so I'm just going to link those for more info. Also I'm going to start with what XP changed from 2003 both because it makes a good comparison and I'd argue 2000/NT 5.0 is the root of the modern Windows era. Your next sentence after the quote implies you probably won't have a problem with that.

* XP/2003: https://en.wikipedia.org/wiki/Features_new_to_Windows_XP

* 2003R2: https://en.wikipedia.org/wiki/Windows_Server_2003#Windows_Se...

* Vista: https://en.wikipedia.org/wiki/Features_new_to_Windows_Vista

* 2008: https://en.wikipedia.org/wiki/Windows_Server_2008#Features

* 7: https://en.wikipedia.org/wiki/Features_new_to_Windows_7

* 2008R2: https://en.wikipedia.org/wiki/Windows_Server_2008_R2#New_fea...

* 8: https://en.wikipedia.org/wiki/Features_new_to_Windows_8

* 2012: https://en.wikipedia.org/wiki/Windows_Server_2012#Features

* 8.1: https://en.wikipedia.org/wiki/Windows_8.1#New_and_changed_fe...

* 2012R2: https://en.wikipedia.org/wiki/Windows_Server_2012_R2#Feature...

* 10: https://en.wikipedia.org/wiki/Features_new_to_Windows_10

* 2016: https://en.wikipedia.org/wiki/Windows_Server_2016#Features

* 2019: https://en.wikipedia.org/wiki/Windows_Server_2019#Features

* 2022: https://en.wikipedia.org/wiki/Windows_Server_2022#Features

* 11: https://en.wikipedia.org/wiki/Features_new_to_Windows_11

* 2025: https://learn.microsoft.com/en-us/windows-server/get-started...

Obviously some of this will be "fluff" and that's up to your own personal definitions, but to act like there haven't been significant changes in every major revision is just nonsense.

Re: No More Blue Fridays

#216

Post by whynotminot »

hello_moto wrote: post #211
whynotminot wrote: post #204

Earlier quoted context omitted.

> You have to ask the customer if they're okay with that citing "our software might failed and brick your machine". I think you’re still missing the point of Canary deployments. The question your sales team should ask is “would you like a 5% chance of a bug harming your system, or a 100% chance?” > It's like: previously Cybersecurity vendors are shy to ask customers to setup Canary systems because that's just "one-mo…

If you refer Canary deployment as the vendor's internal deployment? I definitely agree. What I find it hard is those in Software that suggested to roll it to a few customers first because this isn't cloud deployment doing A/B test when it comes to Virus Definition. Customers must know what's going on when it comes to virus definition and the implication of them whether they're being part of the rollout group or not.

> If you refer Canary deployment as the vendor's internal deployment? I definitely agree.

No, I’m talking about external deployment to customers. They clearly also had a massive failure in their internal processes too, since a bug this egregious should never make it to the release stage. But that is not what I am talking about right now.

> What I find it hard is those in Software that suggested to roll it to a few customers first because this isn't cloud deployment doing A/B test when it comes to Virus Definition.

I don’t care what you’re releasing to customers— application binary, configuration change, virus definition, etc, if it has the chance of doing this much damage it must be deployed in a controlled, phased way. You cannot 100% one-shot deploy any change that has the potential to boot-loop a massive amount of systems like this. This current process is unacceptable.

> Customers must know what's going on when it comes to virus definition and the implication of them whether they're being part of the rollout group or not.

Who says they don’t have to know? Telling your customers that an update is planned and giving them a time window for their update seems reasonable to me.

Re: No More Blue Fridays

#217

Post by tptacek »

Analemma_ wrote: post #210
wongarsu wrote: post #188

Earlier quoted context omitted.

Microsoft has invested in solving this for at least two decades, probably longer. They are just using a different (arguably worse) approach to this than the Unix world. In Windows 9x anti-malware would just run arbitrary code in the kernel that hooked whatever it wanted. In Windows XP a lot of these things got proper interfaces (like the file system filter drivers to facilitate scanning files before they are accessed…

This is a weird reading of history. Microsoft has spent tons of effort getting as much code out of the kernel as possible: Windows drivers used to be almost all kernel-mode, now they're nearly all in userspace and you almost never need to write a kernel-mode Windows driver unless you're doing something with deep OS hooks (like CS was, although apparently even that wasn't actually necessary). The safeguards on kernel…

A thing I think a lot of people don't include in their premises about Crowdstrike is that they're probably the most significant aftermarket endpoint security product in the world (they are what Norton and McAfee were in 2000), which means they're more than large enough for malware to target their code directly, which creates interesting constraints for where their code can run.

I'm not saying I'd run it (I would not), just that I can see why they have a lot of kernel-resident code.

Re: No More Blue Fridays

#218

Post by hello_moto »

whynotminot wrote: post #216
hello_moto wrote: post #211

Earlier quoted context omitted.

If you refer Canary deployment as the vendor's internal deployment? I definitely agree. What I find it hard is those in Software that suggested to roll it to a few customers first because this isn't cloud deployment doing A/B test when it comes to Virus Definition. Customers must know what's going on when it comes to virus definition and the implication of them whether they're being part of the rollout group or not.

> If you refer Canary deployment as the vendor's internal deployment? I definitely agree. No, I’m talking about external deployment to customers. They clearly also had a massive failure in their internal processes too, since a bug this egregious should never make it to the release stage. But that is not what I am talking about right now. > What I find it hard is those in Software that suggested to roll it to a few cu…

If it's virus defn, what's the process here?

* 0day is happening

* Cybersecurity vendors preparing virus definition

* Vendors send update => new virus definition is about to go down in 1 hour, get ready.

Folks are asleep, nobody reads it?

Let's say now let's do Canary: let's deploy to a few customers (this is unclear how this started: should this be opt-in? opt-out?)

Some customers got it, others... who knows, unclear what the processes are here.

Between here and there, 0day exploited customers because AV defn is not there. What now?

I'm not sure how this plays out tbh.

Re: No More Blue Fridays

#219

Post by kayo_20211030 »

enragedcacti wrote: post #185
kayo_20211030 wrote: post #15

This isn't right. If I need a system to run with a piece of code, then it shouldn't run at all if that piece of code is broken. Ignoring the failure is perverse. Let's say that the driver code ensures that some medical machine has safety locks (safeguards) in place to make sure that piece of equipment won't fry you to a crisp; I'd prefer that the whole thing not run at all rather than blithely operate with the safegu…

I agree that some system components should be treated as critical no matter what, but the software at issue in this case (Falcon Sensor or Antivirus more generally) is precautionary and only best effort anyways. I would wager the vast majority of the orgs affected on Friday would have preferred the marginally increased risk of a malware attack or unauthorized use over a 24 hour period instead of the total IT collapse…

I'm with you. What's critical, and what's not? Is it a big thing, or not a big thing? Is this particular machine more critical than the one over there? Security systems need to be at the lowest level, or else some shifty bastard will find a path around them. If it's at the lowest level, the downside of a failure is catastrophic, as we experienced last Friday. The carnage here is ultimately on CrowdStrike. The testing must have been slapdash at best, and missing at worst. eBPF changes nothing. The question is: should we fail, or carry on? eBPF doesn't help with that decision, it only determines the outcome from a system perspective. Any decision is a value judgement; it might be right or wrong, and its outcome either benign or deadly. Choices!

Re: No More Blue Fridays

#220

Post by kayo_20211030 »

ChrisMarshallNY wrote: post #124
kayo_20211030 wrote: post #15

This isn't right. If I need a system to run with a piece of code, then it shouldn't run at all if that piece of code is broken. Ignoring the failure is perverse. Let's say that the driver code ensures that some medical machine has safety locks (safeguards) in place to make sure that piece of equipment won't fry you to a crisp; I'd prefer that the whole thing not run at all rather than blithely operate with the safegu…

> Ignoring the failure is perverse. If the failed system is a security module, I think that's absolutely correct. If the system runs, without the security module, well, that's like forgetting to pack condoms on Shore Leave. You'll likely be bringing something back to the ship with you. Someone needs to be testing the module, and the enclosing system, to make sure it doesn't cause problems. I suspect that it got a gre…

Interesting analogy, but yes. If the module *is* necessary, well, it's necessary and nothing should work without it. Testing must have been a mess here.
Post reply on HN