Live data from Hacker News

No More Blue Fridays

brendangregg.com

141–150 of 282 posts

Re: No More Blue Fridays

#141

eBPF == extended Berkeley Packet Filter https://en.wikipedia.org/wiki/Berkeley_Packet_Filter

Thanks! This was not a familiar acronym to me... and after some digging[0] apparently it's no longer an acronym:

"BPF originally stood for Berkeley Packet Filter, but now that eBPF (extended BPF) can do so much more than packet filtering, the acronym no longer makes sense. eBPF is now considered a standalone term that doesn’t stand for anything."

[0] https://ebpf.io/what-is-ebpf/

Re: No More Blue Fridays

#142

The blog post says: > eBPF, which is immune to such crashes. I tried to Google about this, but I cannot find anything definitive. It looks like you can still break things. Can an expert on eBPF please comment on this claim? This is the best that I could find: https://stackoverflow.com/questions/70403212/why-is-ebpf-sai...

eBPF programs cannot crash the kernel, assuming there are no bugs in the eBPF verifier. There have been such bugs in the past but they seem to be getting more and more rare.

Or in other parts of the kernel. It's been the case in multiple occasions that buggy locking (or more generalised, missing 'resource' release) has caused problems for perfectly safe BPF programs. For example, see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1033398 and the fix https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Re: No More Blue Fridays

#143

Earlier quoted context omitted.

Apple took the lead on this front. It has closed easy access to the kernel by apps, and made a list of APIs to try and replace the lost functionality. Anyone maintaining a kernel module on macOS is stuck in the past. Of course, the target area of macOS is much smaller than Windows, but it is absolutely possible to kick all code, malware and parasitic security services alike, from accessing the kernel. The safest kern…

I don't think Microsoft has a choice with regards to kernel access. Hell, individuals currently use undocumented NT APIs. I can't imagine what happens to backwards compat if kernel access is closed. Apple's closed ecosystem is entirely different. They'll change architectures on a whim and users will go with the flow (myself included).

But Apple doesn’t have the industrial and commercial uses that Linux and Windows have. Where you can’t suddenly switch out to a new architecture without massive amounts of validation costs.

At my previous job they used to use Macs to control scientific instrumentation that needed a data acquisition card. Eventually most of the newer product lines moved over to Windows but one that was used in a validated FDA regulated environment stayed on the Mac. Over time supporting that got harder and harder: they managed through the PowerPC to Intel transition but eventually the Macs with PCIe slots went away. I think they looked at putting the PCIe card in a Thunderbolt enclosure. But the bigger problem is guaranteeing supply of a specific computer for a reasonable amount of time. Very difficult to do these days with Macs.

Re: No More Blue Fridays

#144

The blog post says: > eBPF, which is immune to such crashes. I tried to Google about this, but I cannot find anything definitive. It looks like you can still break things. Can an expert on eBPF please comment on this claim? This is the best that I could find: https://stackoverflow.com/questions/70403212/why-is-ebpf-sai...

eBPF programs cannot crash the kernel, assuming there are no bugs in the eBPF verifier. There have been such bugs in the past but they seem to be getting more and more rare.

This isn't really true. eBPF programs in Linux have access to a large set of helper functions written in plain C. https://lwn.net/Articles/856005/

Re: No More Blue Fridays

#145

Earlier quoted context omitted.

Any industry that wants to reliably deliver software that doesn’t brick systems at scale? I’m confused by your question. Are you telling me the cybersecurity scene is special and shouldn’t follow best practices for software deployment?

Canary deployment for subset of Salesforce customers won't see much of revolt from customers compare to AV definition rollout (not software, but AV definition) in Cybersecurity where gaps between 0day and rollout means you're exposed. If customers found out that some are getting roll out faster than the others, essentially splitting the group into 2, there will be a need for customer opt-in/opt-out. If everyone is op…

Canary deployment doesn’t necessarily mean massive gaps between deployment waves. You can fast-follow. Sure, there may be scenarios with especially severe vulnerabilities where time is of the essence. I’m out of the loop if this crowdstrike update was such a scenario where best practices for software deployment were worth bypassing.

If this is just how they roll with regular definition updates, then their deployment practices are garbage and this kind of large scale disaster was inevitable.

Re: No More Blue Fridays

#146

Earlier quoted context omitted.

> If I were the customers and I found out that I was used as test subject, how would I feel? In reality, every business has relationships that it values more than others. If I wasn't paying a lot for it, and if I was running something that wasn't critical (like my side project) then why not? You can price according to what level of service you want to provide.

Customers will ask to opt-out.

Customers will pay to opt out.

Re: No More Blue Fridays

#147

Earlier quoted context omitted.

If the verifier allowed a Turing-complete language, it would solve the halting probem, which is impossible.

My point is that the verifier could be relaxed to accept programs that never halt, thus not needing to solve the halting problem. You could then have the kernel just kill it after running over a certain maximum amount of time.

Why do you think the kernel crashes when crowdstrike attempts to reference some unavailable address (or whatever it does) instead of just denying that operation and continuing on? That would be the solution using this philosophy "just kill long running program". And no need for eBPF or anything complicated. But it doesn't work that way in practice.

This is just such a naive view. "We can prevent programs from crashing by just taking care to stop them when they do bad things". Well, sure, that's why you have a kernel and userland. But it turns out, some things need to run in the kernel. Or "just deny permission". Then it turns out some programs need to run as admin. And so on.

There is a generality in the halting problem, and saying "we'll just kill long runing programs" just misses the point entirely.

Likely what will happen is that you will kill useful long-running programs, then an exception mechanism will be invented so some programs will not be killed, because they need to run longer, then one of those programs will go into an infinite loop despite all your mechanisms preventing it. Just like the crowdstrike driver managed to bring down the OS despite all the work that is supposed to prevent the entire computer crashing if a single program tries something stupid.

Re: No More Blue Fridays

#148

Earlier quoted context omitted.

I hope though that Microsoft will double down on their eBPF support for Windows after this incident.

Keep in mind they don't just allow any old code to execute in the kernel. They do have rigorous tests (WHQL), it's just Crowdstrike decided that was too burdensome for their frequent updates, and decided to inject code from config files (thus bypassing the control). The fault here is entirely with Crowdstrike.

How rigorous are the tests if faulty data can brick the machine?

Re: No More Blue Fridays

#149

Earlier quoted context omitted.

Canary deployment for subset of Salesforce customers won't see much of revolt from customers compare to AV definition rollout (not software, but AV definition) in Cybersecurity where gaps between 0day and rollout means you're exposed. If customers found out that some are getting roll out faster than the others, essentially splitting the group into 2, there will be a need for customer opt-in/opt-out. If everyone is op…

Canary deployment doesn’t necessarily mean massive gaps between deployment waves. You can fast-follow. Sure, there may be scenarios with especially severe vulnerabilities where time is of the essence. I’m out of the loop if this crowdstrike update was such a scenario where best practices for software deployment were worth bypassing. If this is just how they roll with regular definition updates, then their deployment…

Let's walk this through: Canary deployment to Windows machines. If those Windows machines got hit with BSOD, they will go offline. How do you determine if they go offline because of Canary or because of regular maintenance by the customer's IT cycle?

You can guess, but you cannot be 100% sure.

What if the targeted canary deployments are Employees desktops that are OFFLINE during the time of rollout?

>I’m out of the loop if this crowdstrike update was such a scenario where best practices for software deployment were worth bypassing.

I did post a question: what about other Cybersecurity vendors? Do you think they do canary deployment on their AV definitions?

Here's more context to understand Cybersecurity: https://radixweb.com/blog/what-is-mean-time-to-detect

Cybersecurity companies participate in Sec evaluation annually that evaluates (measure) and grade their performance. That grade is an input for Organizations to select vendors outside their own metrics/measurements.

I don't know if MTTD is included in the contract/SLA. If it does, you got some answer as to why certain decision is made.

It's definitely interesting to see Software developers of HN giving out their 2c for a niche Cybersecurity industry.

Re: No More Blue Fridays

#150
I don't buy it... didn't a bug from RedHat + Crowdstrike have a similar panic issue? I understand in that case it was because of RedHat, but still. I don't think this, by itself will change much.
Post reply on HN