Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No More Blue Fridays

brendangregg.com

Post reply on HN
Print view View original

51–60 of 282 posts

Re: No More Blue Fridays

#51

Post by whynotminot »

hello_moto wrote: post #39
lordnacho wrote: post #11

Earlier quoted context omitted.

They could have helped their luck by doing some of the common sense things suggested in the article. For instance, why not find a subset of your customers that are low risk, push it out to them, and see what happens? Or perhaps have your own fleet of example installations to run things on first. None of which depends on any specific technology.

"find a subset of low risk customers" and use them as test subject? Repeat that a few times to understand the repercussions. If I were the customers and I found out that I was used as test subject, how would I feel?

Canary deployments are already an industry accepted practice and it’s shocking Crowdstrike apparently doesn’t do them.

Re: No More Blue Fridays

#52

Post by kevin_nisbet »

I hate to dispute with someone like Brendan Gregg, but I'm hoping vendors in this space take a more holistic approach to investigating the complete failure chain. I personally tend to get cautious when there is a proposal that x will solve the problem that occurred on y date, especially 3 days after the failure. It may be true, but if we don't do the analysis we could leave ourselves open to blindspots. There may also be plenty of alternative approaches that should be considered and appropriately discarded.

I think the part I specifically dispute is the only negative outcome is wasted CPU cycles. That's likely the case for the class of bug, but there are plenty of failure modes where a bad ruleset could badly brick a system and make it hard to recover.

That's not to say eBPF based security modules isn't the right choice for many vendors, just that let's understand what risks they do and do not avoid, and what part of the failure chain they particularly address.

Re: No More Blue Fridays

#53

Post by School-Cotton »

lucianbr wrote: post #45
skywhopper wrote: post #24

Earlier quoted context omitted.

If the verifier can't determine that the loop will halt, the program is disallowed. Also, if the program gets passed and then runs too long anyway, it's force-halted. So... I guess that solves the halting problem.

So this "solves" the halting problem by creating a new class "might-not-halt-but-not-sure" and lumping it with "does-not-halt". I find it hard to believe the new class is small enough for this to be useful, in the sense that it will avoid all kernel crashes. I rather expect useful or needed code would be rejected due to "not-sure-it-halts", and then people will use some kind of exception or not use the verifier at al…

Well it is useful in practice, there are some pretty useful products based on eBPF on Linux, most notably Cilium (and, shameless plug for the one I’m working on: Parca, an eBPF-based CPU profiler).

Re: No More Blue Fridays

#54

Post by shahahqq »

xyzzy123 wrote: post #40

So many problems though! including commercial monocultures, lack of update consent, blast radius issues, etc etc. There's a commons in our pockets but that is very difficult to regulate for. The will keep putting the gun to your head until you keep choosing the monoculture.

worrisome indeed that now the world knows how many users are affected by crowdstrike so the bad guys just need to poke deeper there

Re: No More Blue Fridays

#55

Post by lucianbr »

efee22 wrote: post #33
skywhopper wrote: post #21

The implicit assumption of the article is that eBPF code can't crash a kernel, but the article itself eventually admits that it can and has done, including last month. eBPF is a safer way of providing kernel-extension functionality, for sure, but presenting it as the perfect solution is just asking to have your argument dismissed. eBPF is not perfect. And there's plenty of things it can't do. The very sandbox rules t…

It's not a silver bullet, however, it is still better to pushing all the panicable bugs into one community-maintained section (e.g. eBPF verifier). All vendors have an incentive to help get right and this is much better than every vendor shipping their own panicable bugs in their own out of tree kernel modules. Additionally, it's not just the industry looking at eBPF, but also academia in terms of formally verifying…

"Improves kernel stability" is great. "Prevents kernel crashes" is a plain lie.

> In the future, computers will not crash due to bad software updates, even those updates that involve kernel code.

Come on. Computers will continue to crash in the future, even when using eBPF. I am quite certain.

Re: No More Blue Fridays

#56

Post by gtsop »

lordnacho wrote: post #11
StevenWaterman wrote: post #6

Earlier quoted context omitted.

If you consider kernel programming to be inherently unsafe, then you would consider this to be inevitable, meaning it's not really the specific company's fault. They were just the unlucky ones.

They could have helped their luck by doing some of the common sense things suggested in the article. For instance, why not find a subset of your customers that are low risk, push it out to them, and see what happens? Or perhaps have your own fleet of example installations to run things on first. None of which depends on any specific technology.

Why even do that? We have virtualization, they could emulate real clients and networks of clients. This particular bug would have been prevented for sure

Re: No More Blue Fridays

#57

Post by jve »

phartenfeller wrote: post #41
kayo_20211030 wrote: post #15

This isn't right. If I need a system to run with a piece of code, then it shouldn't run at all if that piece of code is broken. Ignoring the failure is perverse. Let's say that the driver code ensures that some medical machine has safety locks (safeguards) in place to make sure that piece of equipment won't fry you to a crisp; I'd prefer that the whole thing not run at all rather than blithely operate with the safegu…

The medical machine software should just refuse to run with an error message if a critical driver was not loaded. The OS bricking is causing way more trouble where an IT technician now needs to fix something where it otherwise would just be updating the faulty driver... Also does your car not start if you are missing water for the wiper?

Water for the wiper is userland feature.

3rd party hooking into kernel is 3rd party responsibility. It is like equipping your car with LPG - THAT hooks into engine (kernel). And When I had a faulty gas pressure sensor then my car actually halted (BSOD if you will) instead of automatically failing over to gasoline as it is by design.

You can argue that car had no means to continue execution but kernel has, however invalid kernel state can cause more corruption down the road. Or as parent even points out - carry out lethal doses of something.

Re: No More Blue Fridays

#58

Post by the8472 »

If the filters are loaded at boot and hook into everything then a bug can still lock down the system to a point where it can't be operated or patched anymore (e.g. because you loaded an empty whitelist). So it could end up replacing a boot loop with another form of DoS.

If microsoft includes a hardcoded whitelist that covers some essentials needed for recovery that could make a bug in such a tool easier to fix, but could still cause effective downtimes (system running but unusuable) until such a fix is delivered.

Re: No More Blue Fridays

#59

Post by hiddencost »

pimlottc wrote: post #43
hiddencost wrote: post #25

Earlier quoted context omitted.

I think the article isn't about crowd strike. It's about ebpf.

The second paragraph is 100% about Crowdstrike. It even links to the Wikipedia article: https://en.m.wikipedia.org/wiki/2024_CrowdStrike_incident

CrowdStrike is mentioned, but the goal of the article is to promote eBPF. CrowdStrike is tangentially related because it draws attention to a platform that Gregg has put a lot into.

Re: No More Blue Fridays

#60

Post by twen_ty »

Can someone tell me what's the advantage of eBPF over a user mode driver? The article makes it look it eBPF is have your cake and eat it too solution which is too good to be true? Can you run graphics drivers in eBPF for example?
Post reply on HN