Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

No More Blue Fridays

brendangregg.com

Post reply on HN
Print view View original

41–50 of 282 posts

Re: No More Blue Fridays

#41

Post by phartenfeller »

kayo_20211030 wrote: post #15

This isn't right. If I need a system to run with a piece of code, then it shouldn't run at all if that piece of code is broken. Ignoring the failure is perverse. Let's say that the driver code ensures that some medical machine has safety locks (safeguards) in place to make sure that piece of equipment won't fry you to a crisp; I'd prefer that the whole thing not run at all rather than blithely operate with the safegu…

The medical machine software should just refuse to run with an error message if a critical driver was not loaded. The OS bricking is causing way more trouble where an IT technician now needs to fix something where it otherwise would just be updating the faulty driver... Also does your car not start if you are missing water for the wiper?

Re: No More Blue Fridays

#42

Post by phartenfeller »

blinkingled wrote: post #30

Ok. But the good old push code to staging / canary it before mainstream updates was a simpler way of solving the same problem. Crowdstrike knows the computers they're running on, it is trivial to implement a system where only few designated computers download and install the update and report metrics before the update controller decides to push it to next set.

Why trust somebody else not messing up? With that in place for windows and crowdstrike billions of dollars would be saved and many lives not negatively impacted ...

Re: No More Blue Fridays

#43

Post by pimlottc »

hiddencost wrote: post #25
asynchronous wrote: post #5

Is there a reason for the lack of naming+shaming Crowdstrike in this blogpost? Was it to not give them any more publicity, good or bad?

I think the article isn't about crowd strike. It's about ebpf.

The second paragraph is 100% about Crowdstrike. It even links to the Wikipedia article:

https://en.m.wikipedia.org/wiki/2024_CrowdStrike_incident

Re: No More Blue Fridays

#44

Post by roywiggins »

atrus wrote: post #27
shrx wrote: post #18

From the article: > If the verifier finds any unsafe code, the program is rejected and not executed. The verifier is rigorous -- the Linux implementation has over 20,000 lines of code [0] -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). [0] links to https://github.com/torvalds/linux/blob/master/kernel/bpf/ver... which has this inter…

The halting problem is exhaustive, there isn't an algorithm that is valid for all programs. You can still check for some kinds of infinite loops though!

More specifically, you can accept a set of programs that you are certain do halt, and reject all others, at the expense of rejecting some that will halt. As long as that set is large enough to be practical, the result can be useful. If you eg forbid code paths that jump "backwards", you can't really loop at all. Or require loops to be bounded by constants.

Re: No More Blue Fridays

#45

Post by lucianbr »

skywhopper wrote: post #24
shrx wrote: post #18

From the article: > If the verifier finds any unsafe code, the program is rejected and not executed. The verifier is rigorous -- the Linux implementation has over 20,000 lines of code [0] -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). [0] links to https://github.com/torvalds/linux/blob/master/kernel/bpf/ver... which has this inter…

If the verifier can't determine that the loop will halt, the program is disallowed. Also, if the program gets passed and then runs too long anyway, it's force-halted. So... I guess that solves the halting problem.

So this "solves" the halting problem by creating a new class "might-not-halt-but-not-sure" and lumping it with "does-not-halt". I find it hard to believe the new class is small enough for this to be useful, in the sense that it will avoid all kernel crashes.

I rather expect useful or needed code would be rejected due to "not-sure-it-halts", and then people will use some kind of exception or not use the verifier at all, and then we are back to square one.

Re: No More Blue Fridays

#46

Post by brendangregg »

mrpippy wrote: post #31

> Once Microsoft's eBPF support for Windows becomes production-ready, Windows security software can be ported to eBPF as well. This doesn’t seem grounded in reality. If you follow the link to the “hooks” that Windows eBPF makes available [1], it’s just for incoming packets and socket operations. IOW, MS is expecting you to use the Berkeley Packet Filter for packet filtering. Not for filtering I/O, or object creation/…

Yes, we know eBPF must attach to equivalent events to Linux, but given there are already many event sources and consumers in Windows, the work is to make eBPF another consumer -- not to invent instrumentation frameworks from scratch.

Just to use an analogy: Imagine people do their banking on JavaScript websites with Google Chrome, but if they use Microsoft Edge it says "JavaScript isn't supported, please download and run this .EXE". I'm not sure we'd be asking "if" Microsoft would support JavaScript (or eBPF), but "when."

Re: No More Blue Fridays

#47

Post by School-Cotton »

shrx wrote: post #18

From the article: > If the verifier finds any unsafe code, the program is rejected and not executed. The verifier is rigorous -- the Linux implementation has over 20,000 lines of code [0] -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). [0] links to https://github.com/torvalds/linux/blob/master/kernel/bpf/ver... which has this inter…

eBPF is not Turing complete. Writing it is very annoying compared to writing normal C code for exactly this reason.

Re: No More Blue Fridays

#48

Post by Retr0id »

shrx wrote: post #18

From the article: > If the verifier finds any unsafe code, the program is rejected and not executed. The verifier is rigorous -- the Linux implementation has over 20,000 lines of code [0] -- with contributions from industry (e.g., Meta, Isovalent, Google) and academia (e.g., Rutgers University, University of Washington). [0] links to https://github.com/torvalds/linux/blob/master/kernel/bpf/ver... which has this inter…

The halting problem cannot be solved in the general case, but in many cases you can prove that a program halts. eBPF only allows verifiably-halting programs to run.

Re: No More Blue Fridays

#49

Post by tantalor »

Archelaos wrote: post #37
blinkingled wrote: post #30

Ok. But the good old push code to staging / canary it before mainstream updates was a simpler way of solving the same problem. Crowdstrike knows the computers they're running on, it is trivial to implement a system where only few designated computers download and install the update and report metrics before the update controller decides to push it to next set.

It would mitigate the problem, but not solve it. You can still imagine a condition that only occurs after the update has been rolled out everywhere. Furthermore, such a bug would still be extremely problematic for the concerned customers, even if not all of them were affected. In addition, it would be necessary to react very quickly in the case of zero-day vulnerabilities.

(semantic argument warning)

"Mitigation" is dealing with an outage/breakage after it occurs, to reduce the impact or get system healthy again.

You're talking about "prevention" which keeps it from happening at all.

Canarying is generic approach to prevention, and should not be skipped.

Avoiding the risk entirely (eBPF) would also help prevent outage, but I think we're deluding ourselves to say it "solves" the problem once and for all; systems will still go down due to bad deploys.

Re: No More Blue Fridays

#50

Post by lucianbr »

skywhopper wrote: post #21

The implicit assumption of the article is that eBPF code can't crash a kernel, but the article itself eventually admits that it can and has done, including last month. eBPF is a safer way of providing kernel-extension functionality, for sure, but presenting it as the perfect solution is just asking to have your argument dismissed. eBPF is not perfect. And there's plenty of things it can't do. The very sandbox rules t…

It's casually claiming to have solved the halting problem, at least within some limited but useful context. That should be impossible, and it turns out, it is.

I expect it can be solved within some limited contexts, but those contexts are not useful, at least not at the level of "generic kernel code".

Post reply on HN