Live data from Hacker News

NUMA Siloing in the FreeBSD Network Stack [pdf]

people.freebsd.org

21–30 of 137 posts

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#21
post #2

Is it still accurate that for network bound servers that FreeBSD still outperforms Linux?

I think once you get to a certain traffic level you are forced to do kernel-bypass stuff like DPDK. Regardless of Linux or FreeBSD being the kernel.

Author here.. We're doing ~200Gb/s through the kernel.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#22
post #5

I assumed these were cache servers that went into ISPs, does anyone know what they mean by "Increases AWS cloud management overhead" on a couple of the slides?

Guessing that they have managed services in aws for maintaining all the nodes and where they are. If each box has 2x the ips, they would need more mgmt resources to keep track of what is on that box, if it's online, stats, etc.

Yes, exactly. When we first started the project, I was told that multiple IPs were not an option for that reason.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#24

Earlier quoted context omitted.

I think once you get to a certain traffic level you are forced to do kernel-bypass stuff like DPDK. Regardless of Linux or FreeBSD being the kernel.

Author here.. We're doing ~200Gb/s through the kernel.

That's why I carefully inserted my weasel words of "I think" ;)

Thank you for your contributions. Great information.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#25
post #2

Is it still accurate that for network bound servers that FreeBSD still outperforms Linux?

Author here.. It depends heavily on the workload. We have man years of optimizations for our CDN workload that we have been upstreaming to FreeBSD. See, for example, the other talk I co-presented regarding Kernel TLS on FreeBSD.

https://people.freebsd.org/~gallatin/talks/euro2019-ktls.pdf

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#26
post #15

Isn't it great that we work in an industry where a company like Netflix, which has increasing competition lately, just shares potential competitive advantages like this? I get that, in part, this is a way to get people excited to work for Netflix, however, the people working on this are, probably, pretty proud of what they achieved and thus like to share it with us, their colleagues at other places of work and they h…

Realistically, Disney, or whomever competition, is not going to have enough technical expertise to deploy FreeBSD.

The BSD logo was created by one of Pixar's founders[1], and Disney now owns Pixar. Disney also has money. I'm sure deploying FreeBSD is well within their reach.

1) https://www.freebsd.org/copyright/daemon.html

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#27
post #16

Earlier quoted context omitted.

I think once you get to a certain traffic level you are forced to do kernel-bypass stuff like DPDK. Regardless of Linux or FreeBSD being the kernel.

If it's just serving files you don't necessarily need DPDK/XDP. For server-grade hardware there now is P2P-DMA and TLS accelerators which can offload everything to peripherials while still using normal socket APIs. You get NVMe -(PCIe)-> crypto accelerator -(PCIe)-> ethernet for the bulk of the data. Neither CPU nor main memory see any of the network packets as long as they stay on the happy path. Only connection set…

In the work we've done, the TLS crypto of bulk data is handled entirely in the CPU by the kernel via ktls. Please see the slides, specifically the data flow diagrams.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#28
I wonder how this would compare with Receive Side Scaling (RSS), which you can use to pin NIC queues to cpus, and then pin the rest of the handling to the same cpu, avoiding a significant amount of interprocess communication. NUMA concerns may be more important than CPU pining within a domain though.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#29
post #15

Isn't it great that we work in an industry where a company like Netflix, which has increasing competition lately, just shares potential competitive advantages like this? I get that, in part, this is a way to get people excited to work for Netflix, however, the people working on this are, probably, pretty proud of what they achieved and thus like to share it with us, their colleagues at other places of work and they h…

Realistically, Disney, or whomever competition, is not going to have enough technical expertise to deploy FreeBSD.

... You don't think Disney, a company with a revenue of roughly 60 billion dollars, can hire people with enough technical expertise to deploy FreeBSD?

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#30
post #16

Earlier quoted context omitted.

If it's just serving files you don't necessarily need DPDK/XDP. For server-grade hardware there now is P2P-DMA and TLS accelerators which can offload everything to peripherials while still using normal socket APIs. You get NVMe -(PCIe)-> crypto accelerator -(PCIe)-> ethernet for the bulk of the data. Neither CPU nor main memory see any of the network packets as long as they stay on the happy path. Only connection set…

In the work we've done, the TLS crypto of bulk data is handled entirely in the CPU by the kernel via ktls. Please see the slides, specifically the data flow diagrams.

I think he's talking about the general case (and since XDP is in the discussion, probably the general case for Linux) and not trying to speak specifically to what Netflix is doing for their CDN.
Post reply on HN