Live data from Hacker News

NUMA Siloing in the FreeBSD Network Stack [pdf]

people.freebsd.org

81–90 of 137 posts

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#81
post #73

Earlier quoted context omitted.

you make it sound altruistic. it’s to avoid paying ISPs

It's to avoid paying last mile ISPs who have turned their monopolies into a mafia-like shakedown scheme.

Those last mile ISPs have to install one of these boxes to see a benefit.

Netflix still has to work things out and get the last mile ISPs to play ball ($$$).

But this potentially saves Netflix a whole ton on transit.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#82
post #70

Earlier quoted context omitted.

What made Netflix pick FreeBSD over Linux in the first place?

BSDs unwavering commitment to reliability. A Linux kernel crashes. A lot. FreeBSD ran cdrom.com for 365+ days without rebooting. Just saying

What's "a lot" here, and are you sure that's still true? Because my suspicion is that it might have been true before FAANG sunk billons of R&D into Linux, but I would be shocked if Linux kernels cash at any rare worth noticing today. Although it's splitting hairs, really; when is the last time you saw any major kernel crash?

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#83

Nice to see AMD replacing Intel, they've gone with EPYC 7551 & 7502P, from 2x Intel “Skylake” / “Cascade Lake” Xeon edit: it looks like they hit 200GB/s with both Intel and AMD

It's disappointing to hear that AMD has not provided adequate performance monitoring tools though. I would have thought by the second generation of Zen that this would be an area that AMD had given attention to.

AMD's engineering team is much smaller, and they've always been behind on performance tools. HPC people have been complaining ever since Opteron. It's probably not going to change.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#84
post #79

Earlier quoted context omitted.

> ... just shares potential competitive advantages like this? Is it really a competitive advantage though? I would think that their brand and content library are the main ones. A CDN server doesn't really count IMHO: other streaming services can create their own appliances, though I'm guessing they'll use Linux due to its ubiquity. Though as a long-time FreeBSD fan and part-time user, I'm happy for the patches. I run…

> Is it really a competitive advantage though? It totally is. These things get installed into ISP racks. ISPs have limited space and power for these things. One box pushing 200Gbps in the same space is twice as good (although, note they have 4x100Gbps NICs in this generation, there's room for more perf here). Reducing the number of nodes helps with management of the network as well.

It's funny to think that pushing 196gbps is possibly just as little as ~8000 simultaneous streams.

Also a shame that it is TLS'd. The content already has its own DRM/encryption.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#85
post #61
post #47

Are those changes in 12.1? If not, what would be a time frame?

Not all of them are landed in CURRENT yet, much less stable/12. They'll be in 13.0; I can't speak to any future stable 12.2. On the other hand, CURRENT is pretty solid. A lot of folks, Netflix included, just run CURRENT.

Wait, they run CURRENT in production? Is that... safe?

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#86
post #84
post #79

Earlier quoted context omitted.

> Is it really a competitive advantage though? It totally is. These things get installed into ISP racks. ISPs have limited space and power for these things. One box pushing 200Gbps in the same space is twice as good (although, note they have 4x100Gbps NICs in this generation, there's room for more perf here). Reducing the number of nodes helps with management of the network as well.

It's funny to think that pushing 196gbps is possibly just as little as ~8000 simultaneous streams. Also a shame that it is TLS'd. The content already has its own DRM/encryption.

TLS plays a different role. It makes sure you're receiving the Netflix's DRMed and encrypted content, and not some MITMed content that also happens to be correctly DRMed and encrypted, but contains something else.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#87
post #60

Earlier quoted context omitted.

I know Chelsio has crypto directly on the NIC, but are dedicated crypto accelerator cards a thing and are they ever worth it? Why leave the CPU idle when the CPU itself is a good crypto accelerator (AES-NI, ARMv8 crypto)? AMD Ryzen has a built-in crypto "decelerator" — a FreeBSD driver was written for the crypto engine, but it's disabled by default because it made everything slower than AES-NI. (Though I guess it wou…

Intel has a product line called QAT ("QuickAssisT"?) that does crypto acceleration, as well as compression. I don't know how performant it is. There are definitely several older crypto accelerators that were faster than CPUs of the time; I don't know if any of them (outside of QAT) is still relevant. The AMD Zen1 Crypto Co-Processor is indeed slower than AESNI; I think it's mostly used by stuff like SecureBoot, TPM,…

Supposedly QAT is built into the chipsets of Skylake and above now. I've never seen anyone try it, though.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#88
post #16

Earlier quoted context omitted.

If it's just serving files you don't necessarily need DPDK/XDP. For server-grade hardware there now is P2P-DMA and TLS accelerators which can offload everything to peripherials while still using normal socket APIs. You get NVMe -(PCIe)-> crypto accelerator -(PCIe)-> ethernet for the bulk of the data. Neither CPU nor main memory see any of the network packets as long as they stay on the happy path. Only connection set…

I know Chelsio has crypto directly on the NIC, but are dedicated crypto accelerator cards a thing and are they ever worth it? Why leave the CPU idle when the CPU itself is a good crypto accelerator (AES-NI, ARMv8 crypto)? AMD Ryzen has a built-in crypto "decelerator" — a FreeBSD driver was written for the crypto engine, but it's disabled by default because it made everything slower than AES-NI. (Though I guess it wou…

From what I recall, the chelsio cards only support a mode of encryption suitable for storage devices, and it's not something you'd use for streaming media.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#89
post #71

The Linux network stack has been the bane of my existence trying to squeeze more performance out of KeyDB. I really hope it gets this kind of love in the future. On Linux there’s a spinlock in do soft_irq that blocks even in non-blocking IO.

Why not use DPDK? All of these problems go away, and people have reported hitting 1Tbps on a single node.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#90
Lucky you, still living in the 90s.

In our timeline, things are not looking that great. There was a big terrorist attack on US soil in 2001 and a series of wars in the middle east. We have computers that fit in your pocket. Scientists discovered something called the Higgs Boson and landed a probe on a comet. Microsoft released a lot of their code as open source. And Linux has achieved almost total supremacy over the server, mobile and supercomputer markets.

Post reply on HN