Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An eBPF loophole: Using XDP for egress traffic

loopholelabs.io

Post reply on HN
Print view View original

41–50 of 81 posts

Re: An eBPF loophole: Using XDP for egress traffic

#41

Post by cptnntsoobv »

shivanshvij wrote: post #21
drewg123 wrote: post #14

Earlier quoted context omitted.

I come from a very different world (optimizing the FreeBSD kernel for the Netflix CDN, running on bare metal) but performance leaps like this are fascinating to me. One of the things that struck me when reading this with only general knowledge of the linux kernel is: What makes things so terrible? Is iptables really that bad? Is something serialized to a single core somewhere in the other 3 scenarios? Is the CPU at 1…

As far as we can tell, it’s a mixture of a lot of things. One of the questions I got asked was how useful this is if you have a smaller performance requirement than 200Gbps (or, maybe a better way to put it, what if your host is small and can only do 10Gbps anyways). You’ll have to wait for the follow up post with the CNI plugin for the full self-reproducible benchmark, but on a 16 core EC2 instance with a 10Gbps con…

> IPtables couldn’t do more than 5Gbps of throughput (TCP!)

Is this for a single connection? IIRC, AWS has a 5gbps limit per connection, does it not? I am guessing since you were able to get to ~10 it must be a multi connection number.

Re: An eBPF loophole: Using XDP for egress traffic

#42

Post by shivanshvij »

cptnntsoobv wrote: post #39

XDP, and the eBPF ecosystem in general, is quite neat. However, a word of caution: * The BPF verifier's DX is not great yet. If it finds problems with your BPF code it will spit our a rather inscrutable set of error messages that often requires a good understanding of the verifier internals (e.g the register nomenclature) to debug * For the same source code, the code generated by the verifier can change across compil…

We absolutely ran into these issues.

A couple notes that help quite a bit:

1. Always build the eBPF programs in a container - this is great for reproducibility of course, but also makes DevX on MacOS better for those who prefer to use that.

2. You actually can do a full checksum! You need to limit the MTU but you can:

  static __always_inline void tcp_checksum(const struct iphdr *ip_header, struct tcphdr *tcp_header, const __u16 tcp_len, const void *data_end) {
    __u32 sum = 0;
    __u16 *buf = (void *)tcp_header;
    ip_header_pseudo_checksum(ip_header, tcp_len, &sum);
    tcp_header->check = 0;
    __u16 max_packet_size = tcp_len;
    if (max_packet_size > MAX_TCP_PACKET_SIZE) {
        max_packet_size = MAX_TCP_PACKET_SIZE;
    }
    for (int i = 0; i  data_end) {
            break;
        }
        sum += *buf;
        buf++;
    }
    if ((void *)buf + 1 check = csum_fold_helper(sum);
  }
With that being said, it's not lost on me that XDP in general is something you should only reach for once you hit some sort of bottleneck. The original version of our network migration was actually implemented in userspace for this exact reason!

Re: An eBPF loophole: Using XDP for egress traffic

#43

Post by shivanshvij »

cptnntsoobv wrote: post #41
shivanshvij wrote: post #21

Earlier quoted context omitted.

As far as we can tell, it’s a mixture of a lot of things. One of the questions I got asked was how useful this is if you have a smaller performance requirement than 200Gbps (or, maybe a better way to put it, what if your host is small and can only do 10Gbps anyways). You’ll have to wait for the follow up post with the CNI plugin for the full self-reproducible benchmark, but on a 16 core EC2 instance with a 10Gbps con…

> IPtables couldn’t do more than 5Gbps of throughput (TCP!) Is this for a single connection? IIRC, AWS has a 5gbps limit per connection, does it not? I am guessing since you were able to get to ~10 it must be a multi connection number.

No this was multiple connections - and we tried with both `iperf2` and `iperf3`, UDP and TCP traffic. UDP actually does much worse on `iptables` than TCP, and I'm not sure why just yet.

Re: An eBPF loophole: Using XDP for egress traffic

#44

Post by mgaunard »

shivanshvij wrote: post #42
cptnntsoobv wrote: post #39

XDP, and the eBPF ecosystem in general, is quite neat. However, a word of caution: * The BPF verifier's DX is not great yet. If it finds problems with your BPF code it will spit our a rather inscrutable set of error messages that often requires a good understanding of the verifier internals (e.g the register nomenclature) to debug * For the same source code, the code generated by the verifier can change across compil…

We absolutely ran into these issues. A couple notes that help quite a bit: 1. Always build the eBPF programs in a container - this is great for reproducibility of course, but also makes DevX on MacOS better for those who prefer to use that. 2. You actually can do a full checksum! You need to limit the MTU but you can: static __always_inline void tcp_checksum(const struct iphdr *ip_header, struct tcphdr *tcp_header, c…

How do containers help when bpf is mostly a matter of kernel version?

Re: An eBPF loophole: Using XDP for egress traffic

#47

Post by zygentoma »

zygentoma wrote: post #45

The page has no text for me (only the table of contents on the side, that updates by scrolling over the completely blank purple page …) I'm using Firefox

Removing this:

  @layer base {
    :root, #nd-docs-layout {
      --fd-layout-offset: max(calc(50vw - var(--fd-layout-width)/2),0px);
    }
  }
fixes it for me and I can read the text now …

EDIT: Oh, or making the window less than around 3000 px wide does also fix it, and resizing you can see what is happening there …

Re: An eBPF loophole: Using XDP for egress traffic

#48

Post by cptnntsoobv »

shivanshvij wrote: post #42
cptnntsoobv wrote: post #39

XDP, and the eBPF ecosystem in general, is quite neat. However, a word of caution: * The BPF verifier's DX is not great yet. If it finds problems with your BPF code it will spit our a rather inscrutable set of error messages that often requires a good understanding of the verifier internals (e.g the register nomenclature) to debug * For the same source code, the code generated by the verifier can change across compil…

We absolutely ran into these issues. A couple notes that help quite a bit: 1. Always build the eBPF programs in a container - this is great for reproducibility of course, but also makes DevX on MacOS better for those who prefer to use that. 2. You actually can do a full checksum! You need to limit the MTU but you can: static __always_inline void tcp_checksum(const struct iphdr *ip_header, struct tcphdr *tcp_header, c…

> You actually can do a full checksum

Indeed! This is what I had in mind when I wrote "cumbersome" :).

It's been a while for me to be able to recall whether the problem was the verifier or me, and things may have improved since, but I recall having the verifier choke on a static size limit too. Have you been able to use this trick successfully?

> Always build the eBPF programs in a container

That should work generally but watch out for any weirdness due to the fact that in a container you are already inside a couple of layers of networking (bridge, netns etc.).

Re: An eBPF loophole: Using XDP for egress traffic

#49

Post by cptnntsoobv »

shivanshvij wrote: post #43
cptnntsoobv wrote: post #41

Earlier quoted context omitted.

> IPtables couldn’t do more than 5Gbps of throughput (TCP!) Is this for a single connection? IIRC, AWS has a 5gbps limit per connection, does it not? I am guessing since you were able to get to ~10 it must be a multi connection number.

No this was multiple connections - and we tried with both `iperf2` and `iperf3`, UDP and TCP traffic. UDP actually does much worse on `iptables` than TCP, and I'm not sure why just yet.

For UDP I'd look into GSO/GRO to get an upper bound on what pure kernel can do.

With performance benchmarking, specially in networking there is no end to "oh, but did you think of that?!" :)

Post reply on HN