Live data from Hacker News

eBPF Verification Is Untenable

twitter.com

11–20 of 116 posts

Re: eBPF Verification Is Untenable

#11
First off, I kinda skimmed this.

So I think the critical thing here is that verification is not enough. It has to be the critical thing, because the implementation in the kernel might suck but Microsoft has shown that it's possible to build a powerful eBPF verifier that isn't a hacky mess.

The main issue is seemingly these helper functions. The position is that even a perfectly verified program won't be safe because of them. To me, the situation makes me think "so why are we allowing these helper functions?". The suggestion is, among other things, to replace these helpers with Rust code. But couldn't we just have the helpers not suck to begin with?

Using the Rust compiler as a sort of safety oracle also ignores the fact that rustc has numerous problems that can lead to unsafe code without `unsafe` (and tbh I don't really see the project prioritizing these cases because it's just not a meaningful problem for the typical rust threat model). They sort of address this but not very well imo - timers and runtime mitigations aren't ideal.

I think what might make much more sense is to instead have the eBPF Virtual Machine (and verifier) written in Rust, including all helper functions, but to still execute pure, verified ebpf within it, using a verifier that's been built in a way that's actually sound.

1. The verifier attack surface goes down because it's Rust. I think that removes the need to keep it in userland, which would fly for Windows / BSD but not Linux.

2. Helpers are in Rust so they're at least safer - I feel like this addresses a (the?) major priority in the paper. Based on the paper's notes about implementing helpers in rust requiring no unsafe, it's probably safe to say that the verifier and helpers being in Rust would solve a lot of problems without requiring eBPF programs to be in Rust (and good news, Rust programs can expose a C API).

3. We don't throw out the baby with the bath water. A verified program is a cool thing to have. I would rather keep verification.

Re: eBPF Verification Is Untenable

#12

Hm. Doesn’t look viable to me. I’m not against language-based security, proof-carrying code, and all that, but I have less than perfect confidence that the Rust compiler currently is or will soon be sound enough to be secure against actively hostile code—AFAIU the language designers haven’t even written down their core calculus, let alone proven it sound. Putting the entirety of the Rust compiler (including, at least…

> to be secure against actively hostile code Was that a requirement for the predecessor of eBPF: Custom kernel modules?

Kernel modules require root privileges to load and the Linux kernel's philosophy (pre user namespaces lollllll) was that root -> kernel privesc didn't matter.

Re: eBPF Verification Is Untenable

#13
post #3

This is weird. 1. Instead of having the kernel verify the program about to be installed at installation time, they rely on a trusted compiler and having the kernel perform signature validation. This means that the kernel is relying on a userspace component to enforce kernel-level safety guarantees, adds another level of coupling (via key infrastructure) between the kernel and a particular version of the Rust compiler…

Your point 1 is the elephant herd in the room. If I were a paranoid person, I would think it’s by design - build in a way to compromise a system retroactively.

[deleted]

Re: eBPF Verification Is Untenable

#14
This paper is an easy read, but it's basically just restating the premises of eBPF:

* Most programs can't be expressed in verified eBPF.

* The verifier functions, to the extent it does, in large part by rejecting most programs (and implicitly limiting the uses to which eBPF can be put).

* This is "extension code", and by definition, it interacts with the unsafe, unverified C code that the kernel is built out of.

(In addition to helpers, most serious eBPF-based systems also interact extensively with userland code, which is also not verified, and might even be memory-unsafe, though that's increasingly less likely).

It follows from these premises that vendors should be careful about enabling non-root access to eBPF; when you do that, you really are placing a lot of faith in the verifier. And: most people don't allow non-root eBPF. The verifier is in an uncomfortable place between being a security boundary and a reliability tool.

I'd argue that most of the benefit of eBPF is that you're unlikely to panic your kernel playing with it. Ironically, that's a feature you might not get out of signed, userland-verified, memory-safe Rust code.

Re: eBPF Verification Is Untenable

#15

Hm. Doesn’t look viable to me. I’m not against language-based security, proof-carrying code, and all that, but I have less than perfect confidence that the Rust compiler currently is or will soon be sound enough to be secure against actively hostile code—AFAIU the language designers haven’t even written down their core calculus, let alone proven it sound. Putting the entirety of the Rust compiler (including, at least…

> to be secure against actively hostile code Was that a requirement for the predecessor of eBPF: Custom kernel modules?

eBPF is not a replacement for the general concept of custom kernel modules.

Re: eBPF Verification Is Untenable

#17
post #3

This is weird. 1. Instead of having the kernel verify the program about to be installed at installation time, they rely on a trusted compiler and having the kernel perform signature validation. This means that the kernel is relying on a userspace component to enforce kernel-level safety guarantees, adds another level of coupling (via key infrastructure) between the kernel and a particular version of the Rust compiler…

The escape hatches are unfortunately core to how eBPF in the kernel can work at the moment. It keeps the kernel from having to provide every possible piece of data the program might need as an argument, and provides a lot of assists that otherwise wouldn't be verifiable in general code. Stuff like string operations that would cut into the max 4096 instruction count.

Re: eBPF Verification Is Untenable

#18
post #5

I'm not happy about the entire concept of running user code in the kernel. As a special-purpose hack for servers that do very little else, maybe. As a standard OS feature, it seems to create too big an attack surface. One which has been exploited.[1] [1] https://www.theregister.com/2022/02/23/chinese_nsa_linux/

It requires root to use, if someone has root they’ve already owned your system anyway.

Re: eBPF Verification Is Untenable

#19
post #3

This is weird. 1. Instead of having the kernel verify the program about to be installed at installation time, they rely on a trusted compiler and having the kernel perform signature validation. This means that the kernel is relying on a userspace component to enforce kernel-level safety guarantees, adds another level of coupling (via key infrastructure) between the kernel and a particular version of the Rust compiler…

The escape hatches are unfortunately core to how eBPF in the kernel can work at the moment. It keeps the kernel from having to provide every possible piece of data the program might need as an argument, and provides a lot of assists that otherwise wouldn't be verifiable in general code. Stuff like string operations that would cut into the max 4096 instruction count.

For the most part, the kernel doesn't provide generic string processing helpers. Most helpers are there to form the basis for specific eBPF integration points (the majority are packet and socket handling tools).

https://man7.org/linux/man-pages/man7/bpf-helpers.7.html

Re: eBPF Verification Is Untenable

#20
post #3

This is weird. 1. Instead of having the kernel verify the program about to be installed at installation time, they rely on a trusted compiler and having the kernel perform signature validation. This means that the kernel is relying on a userspace component to enforce kernel-level safety guarantees, adds another level of coupling (via key infrastructure) between the kernel and a particular version of the Rust compiler…

Your point 1 is the elephant herd in the room. If I were a paranoid person, I would think it’s by design - build in a way to compromise a system retroactively.

That makes no sense
Post reply on HN