71–80 of 81 posts
What I don’t really understand is why iptables and tv is so slow.
If the kernel can’t route packets at line speed, how are userspace applications saturating it?
Earlier quoted context omitted.
This is why I am always skeptical when anyone writes that they are the first to do something… the added caveat is always, “that we know of”
"Who did it first" is not interesting to me, but what they're doing isn't a "loophole"; that's all I'm concerned with.
"Whoops, sorry, an innocent kernel update broke your entire production!"
Earlier quoted context omitted.
"Who did it first" is not interesting to me, but what they're doing isn't a "loophole"; that's all I'm concerned with.
It's a weird choice of word (especially in the company name, honestly). To my mind, loopholes are things that get closed. I would not want to be relying on a loophole for anything. "Whoops, sorry, an innocent kernel update broke your entire production!"
"Go ahead and do stuff like this and don't worry about whether it's a 'loophole' is I guess my whole point".
Earlier quoted context omitted.
Forgive me of my ignorance, but is XDP faster than DPDK for packet processing? It seems like DPDK has had a lot of work done for hardware optimizations that allow speeds that I can’t recall XDP being able to do. I have not looked too deeply into this though, so I’m very open to being wrong!
DPDK will give you the absolute best performance, period. But it will do so with tradeoffs that are far from negligible, especially on mixed-workload machines like a docker host/k8s node/hypervisor. 1. to get the absolute best performance, you're running in poll-mode, and burning cpu cores just for packet processing 2. the network interface is invisible to the kernel, making non-accelerated traffic on said interface…
XDP (eXpress Data Path) is the fastest packet processing framework in linux - but it only works for incoming (ingress) traffic. We discovered how to use it for outgoing (egress) traffic by exploiting a loophole in how the linux kernel determines packet direction. Our technique delivers 10x better performance than current solutions, works with existing Docker/Kubernetes containers, and requires zero kernel modificatio…
> XDP for Egress Traffic Flow Diagram
1. Are packages still raw application data when they get to veth0-A without going through the tcp-stack?
2. In container to container communication case, do you plan redirect packages directly from veth0-A to veth0-B
Earlier quoted context omitted.
DPDK will give you the absolute best performance, period. But it will do so with tradeoffs that are far from negligible, especially on mixed-workload machines like a docker host/k8s node/hypervisor. 1. to get the absolute best performance, you're running in poll-mode, and burning cpu cores just for packet processing 2. the network interface is invisible to the kernel, making non-accelerated traffic on said interface…
Oh wow interesting, so the rewrite only went 1/2 as fast? I know cloudflare uses ebpf quite heavy whereas the Great Firewall uses DPDK. I wonder if cloudflare's motivation is just to run it easier on GCP. Any cloudflare employee's here?
As for why Cloudflare uses eBPF where the GFW uses DPDK I can see a few reasons:
- DPDK was the only game in town when the GFW started, while eBPF was the hot new thing for Cloudflare's recent endeavors. GFW did not have any choice.
- Cloudflare has a performance focus, but still has a bit of "hardware is cheap, engineers are expensive", making eBPF more than fine.
- The GFW runs on dedicated machines on the traffic path, while I would expect most of Cloudflare's eBPF endeavors run directly on mixed-workloads machines. One of their first blogpost about it (dropping x Mpps) specifically calls the reason was to protect an end machine directly on said machine, by preventing bad packets from reaching the kernel stack
- Most of the operational advantages I already mentioned. GFW is fine with a "drop traffic if DPDK down", but Cloudflare is absolutely not, making the operational simplicity a bit win.
I bet Cloudflare does have quite a hefty DPDK application used for the traffic scrubbing part of their anti-ddos; but they don't publicize it because it's not as shiny as eBPF.
There are also other advantages to eBPF that makes it better suited to a multi-product company like cloudflare that don't weigh as much as in a mono-product org like the GFW. Take for example the much easier testing, dev env on any laptop, ... Or that eBPF probes can be written in Rust, getting the same featureful language to run in the kernel and in userspace (the classic combo is Go in userspace, C in kernelspace).
XDP, and the eBPF ecosystem in general, is quite neat. However, a word of caution: * The BPF verifier's DX is not great yet. If it finds problems with your BPF code it will spit our a rather inscrutable set of error messages that often requires a good understanding of the verifier internals (e.g the register nomenclature) to debug * For the same source code, the code generated by the verifier can change across compil…
On checksums: Incremental updates are the path of least pain only if the packet’s checksum is valid and not CHECKSUM_PARTIAL. With modern offloads (TSO/GSO/GRO/checksum offload), the checksum visible to XDP is often zero/garbage because the NIC fills it later. In practice, either disable offloads for that traffic or recompute from scratch with bpf_csum_diff() plus bpf_l3_csum_replace() / bpf_l4_csum_replace().
The verifier: This is a fun one, when you make a small change and suddenly the verifier won't allow it.
And the moment you start modifying packets too much yourself, you're on the hook for everything the kernel used to do for you.
I once went down the rabbit hole of building a minimal TCP stack, and the experience was exactly as you'd expect. Getting to 95% done felt quick, but that last 5% was a nightmare (if 100% is even achievable)
Earlier quoted context omitted.
Forgive me of my ignorance, but is XDP faster than DPDK for packet processing? It seems like DPDK has had a lot of work done for hardware optimizations that allow speeds that I can’t recall XDP being able to do. I have not looked too deeply into this though, so I’m very open to being wrong!
DPDK will give you the absolute best performance, period. But it will do so with tradeoffs that are far from negligible, especially on mixed-workload machines like a docker host/k8s node/hypervisor. 1. to get the absolute best performance, you're running in poll-mode, and burning cpu cores just for packet processing 2. the network interface is invisible to the kernel, making non-accelerated traffic on said interface…
From 2022: https://www.samd.is/2022/06/13/egress-XDP.html You can also use XDP for outgoing packets for tap interfaces.
Earlier quoted context omitted.
Oh wow interesting, so the rewrite only went 1/2 as fast? I know cloudflare uses ebpf quite heavy whereas the Great Firewall uses DPDK. I wonder if cloudflare's motivation is just to run it easier on GCP. Any cloudflare employee's here?
Pretty much, which was incredible for a half day rewrite, learning ebpf in rust included. The effort to result ratio is simply incredible. A few cleanups and optimizations later and I was pretty much convinced I would not need to touch DPDK again (so was the company). Following this experiments, I wrote some actual production grade eBPF routers at this company that are in production, much more complex, but still able…