Sorry, but neither eBPF nor Rust nor formal verification nor ... is going to solve that problem. Repeat after me: there are no technical solutions to social problems. As long as the result of such an outage is basically a "oh, a software problem! shrug", _nothing_ will change.
No More Blue Fridays
111–120 of 282 posts
Re: No More Blue Fridays
#1121. How does eBPF solve this? It makes it more difficult, sure, but it'll almost always be possible to cause a crash, if you try hard enough.
2. More importantly, the problem is rarely fixable by changing technology, because typically, problems are caused by people and their connections: social/corporate pressures, profit-seeking, mental health being treated as unimportant, et cetera. eBPF can't fix those, and as long as corporations have social structures that penalize thoroughness and caution, and incentivize getting 'the most stuff' done, this will persist as a problem.
Re: No More Blue Fridays
#113Earlier quoted context omitted.
I hope though that Microsoft will double down on their eBPF support for Windows after this incident.
Keep in mind they don't just allow any old code to execute in the kernel. They do have rigorous tests (WHQL), it's just Crowdstrike decided that was too burdensome for their frequent updates, and decided to inject code from config files (thus bypassing the control). The fault here is entirely with Crowdstrike.
Is there any evidence that the config files had arbitrary code in them? The only analysis I'd seen so far indicated a parsing error loading a viral signature database that was routinely updated, but in this case was full of garbage data.
Re: No More Blue Fridays
#114Earlier quoted context omitted.
I hope though that Microsoft will double down on their eBPF support for Windows after this incident.
Doubt it. Microsoft is clearly over Windows. They continue to produce it but every release feels like "Ugh, fine, since you are paying me a ton of money." Internally, Microsoft is running more and more workloads on Linux and externally, I've had .Net team tell me more than once that Linux is preferred environment for .Net. SQL Server team continues to push hard for Linux compatibility with every release. EDIT: Window…
> SQL Server team continues to push hard for Linux compatibility with every release.
It's kinda funny that the DB that was once a fork of Sybase that was ported to Windows is trying to make its way back to Unix.
Re: No More Blue Fridays
#115Earlier quoted context omitted.
Keep in mind they don't just allow any old code to execute in the kernel. They do have rigorous tests (WHQL), it's just Crowdstrike decided that was too burdensome for their frequent updates, and decided to inject code from config files (thus bypassing the control). The fault here is entirely with Crowdstrike.
Is there any evidence that the config files had arbitrary code in them? The only analysis I'd seen so far indicated a parsing error loading a viral signature database that was routinely updated, but in this case was full of garbage data.
Perhaps not verified, but some smart people do have convincing arguments:
Re: No More Blue Fridays
#1161. How does eBPF solve this? It makes it more difficult, sure, but it'll almost always be possible to cause a crash, if you try hard enough. 2. More importantly, the problem is rarely fixable by changing technology, because typically, problems are caused by people and their connections: social/corporate pressures, profit-seeking, mental health being treated as unimportant, et cetera. eBPF can't fix those, and as long…
> it'll almost always be possible to cause a crash, if you try hard enough.
If you think you know a way to crash the Linux kernel by loading and running an eBPF program, you should report a bug.
Re: No More Blue Fridays
#117Earlier quoted context omitted.
Obviously in userspace it would only crash the running program and not the entire operating system? It's a significant difference. All of the service interruptions would have been just "computer temporarily not protected by crowdstrike agent". Not the same thing at all.
> It's a significant difference. When various apps running the world are crashing, unable to execute because malware protection is failing, there is no difference.
_No_ difference oversells it, IMO -- the fact that the entire OS crashed is what made fixing the bug so arduous, since it required in-person intervention. To be sure, running the code in userspace would still cause unacceptable service interruptions, but the fix could be applied remotely.
Re: No More Blue Fridays
#118Earlier quoted context omitted.
At Ring 3 it would crash an app, not the entire OS. Yes, the kernel is fine and is not to blame. But running basically a rootkit controlled by a third party indeed is to blame.
> At Ring 3 it would crash an app, not the entire OS. That's still an outage for those key systems.
It is an outage for the monitoring system, not the system that it monitors.
Re: No More Blue Fridays
#119Can someone tell me what's the advantage of eBPF over a user mode driver? The article makes it look it eBPF is have your cake and eat it too solution which is too good to be true? Can you run graphics drivers in eBPF for example?
AFAIK, an ebpf function can only access memory it got handed as an argument or as result from a very limited number of kernel functions. Your function will not load if you don't have boundary checks. Fighting the ebpf validator is a bit like fighting Rust's borrow checker; annoying, at times it's too conservative and rejects perfectly correct code, but it will protect you from panics. Loops will only be accepted if the validator can prove they'll end in time; this means it can be a pain to make the validator to accept a loop. Also, ebpf is a processor-independent byte code, so vectorizing code is not possible (unless the byte code interpreter itself does it).
Given all its restrictions, I doubt something complex like a graphics driver would be possible. But then, I know nothing about graphics driver programming.
Re: No More Blue Fridays
#120The blog post says: > eBPF, which is immune to such crashes. I tried to Google about this, but I cannot find anything definitive. It looks like you can still break things. Can an expert on eBPF please comment on this claim? This is the best that I could find: https://stackoverflow.com/questions/70403212/why-is-ebpf-sai...
eBPF programs cannot crash the kernel, assuming there are no bugs in the eBPF verifier. There have been such bugs in the past but they seem to be getting more and more rare.