111–120 of 192 posts
Earlier quoted context omitted.
The end goal with bpf is to allow arbitrary untrusted programs to load bpf programs. If you were just loading kernel modules you wouldn't be able to maintain kernel integrity and let arbitrary programs load code.
Loading BPF requires root, so does loading kernel modules. I.e. if you have permissions to load BPF, you can already load arbitrary code.
This looks very similar to webassembly work going on right now, both use a secure VM, and both run in kernel space. Would webassembly be a more general purpose way of accomplishing something like this?
A BPF interpreter can literally be ~100 LoC. A WebAssembly VM on the other hand will likely be ~1million LoC (without checking). One is suitable for embedding into a kernel, the other isn't.
Perhaps WebAssembly kernel-space programming would still be viable for major trusted projects, like Kafka, where you literally dedicate the entire machine to that application.
I have a hard time understanding what you would use it for. I could understand a use-case, but I fail to understand why it would be that much useful. I have a sense it allows much better performance for horizontal scaling, but I'm not sure...
Since extended BPF is in the Linux kernel (and will be in other kernels in the future), everyone is getting it, and we'll see more use cases over time. In some ways it's like the birth of JavaScript for the browser, and all the new applications it made possible. But it goes further than that: we could still analyze and debug JavaScript applications using traditional tools. But BPF programs are neither process-space or kernel routines, and are outside the view of everything. No visibility in ps(1), top(1), or lsmod(8). We're having to create new tools to even see what's running on the CPUs. Every performance monitoring product that shows a process table with CPU consumption will now need a BPF program table as well.
[0] https://cilium.io/ [1] https://github.com/facebookincubator/katran [2] http://www.brendangregg.com/blog/2016-11-30/linux-bcc-tcplif...
Earlier quoted context omitted.
I've thought that many times and always been wrong. At the end of the day, code is just code. After you take the time to understand the environment it's running in, it's not that much different than anything else you could write. Creating BPF in the first place took a lot of cleverness. Figuring out how to fit it into a massive infrastructure at a place like Facebook took a lot of cleverness. But most of the people a…
There is lots of code that is difficult to understand, for anyone. What about code that proves a complex, many page mathematical theorem? What about code that does complex and very math heavy things like GCM encryption modes, or statistical compression via prediction by partial matching and arithmetic coding? Just reading the code isn't always enough, plenty of complicated code requires understanding of concepts far…
The hard part should be understanding the theorem. If you understand that and the code is still difficult, then the code should be better written.
I guess one exception is when you're optimizing for performance and make an explicit decision to sacrifice maintainability.
eBPF can be viewed as a mechanism to safely run user code in kernel since it uses a DSL and a compiler before the byte code is executed in kernel. This opens up doors for running performance critical functionality in kernel without having to bundle it with the kernel or very tightly coupled with the kernel version. Optimizing FUSE is an example: https://extfuse.github.io/ I expect custom security auditing software, r…
eBPF can be viewed as a mechanism to safely run user code in kernel since it uses a DSL and a compiler before the byte code is executed in kernel. This opens up doors for running performance critical functionality in kernel without having to bundle it with the kernel or very tightly coupled with the kernel version. Optimizing FUSE is an example: https://extfuse.github.io/ I expect custom security auditing software, r…
According to https://news.ycombinator.com/item?id=18496054 , these programs have to halt? How does this system guarantee that the programs halt? Does this mean eBPF is not Turing complete?
Yes, yes. It's got some access control and security sugar sprinkled in but the idea of 'userland for everything!' is clearly wrong if eBPF moving stuff back into the kernel is "The biggest OS development in my career"
Earlier quoted context omitted.
Real-time, low latency, network-based applications. At the pace of network events, CPU is still very fast by perhaps at least order of magnitude. However, latency introduced by system calls is significant. This allows you to run certain classes of application in kernel space with these overheads largely mitigated. Principally it's monitoring and "observability" applications, but apparently it's much more flexible now…
To get the realtime benefits I'd have to run everything that requires realtime scheduling as an eBPF program, right? I mean, suppose I want to make an eBPF program to pipe audio input into my very fancy DSP algo. Awesome. But now I've limited the remainder of my signal chain to remain in eBPF land, haven't I? The moment I throw the signal to Supercollider, Pd, some Jack client, etc., I'm back to being a mere userland…
Right, because jack is a user space collection of programs. It's obvious that any real-time or low latency DSP audio processing pipeline would have to be moved to BPF in its entirety.
So yes, BPF would work just fine if a new DAW pipeline was designed to work with in it. User space programs would only function as GUI front ends to load and configure BPF audio modules. This way audio can be sourced or sinked via DAC/ADC devices, disk, and network devices without the user-kernel space latencies.
Off topic: this is the same Brendan Gregg of flame charts fame [0][1]. It has solved my skin quite a few times when trying to figure out performance bottlenecks in Python apps (using pyflame[2] to capture data and FlameGraph[1] to convert it to displayable SVG). [0] http://www.brendangregg.com/flamegraphs.html [1] https://github.com/brendangregg/FlameGraph [2] https://github.com/uber-archive/pyflame
I've had success using `py-spy` for debugging perf issues. Flamegraphs are much nicer to work with than cProfile's output.