NUMA Siloing in the FreeBSD Network Stack [pdf] • Hacker News Board

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#91

Post by mlyle » Mon, Sep 23, 2019, 5:00 AM UTC

nine_k wrote: post #86

mlyle wrote: post #84

Earlier quoted context omitted.

It's funny to think that pushing 196gbps is possibly just as little as ~8000 simultaneous streams. Also a shame that it is TLS'd. The content already has its own DRM/encryption.

TLS plays a different role. It makes sure you're receiving the Netflix's DRMed and encrypted content, and not some MITMed content that also happens to be correctly DRMed and encrypted, but contains something else.

Easy enough to make the content authenticate itself, instead of repeating cryptographic work for each viewer.

E.g. every 64k block contains a SHA256 hash of the next 64k block. When a user seeks, you provide the hash of the first block they'll be receiving over a trusted channel. (It already can exist at start of stream RSA signed or whatever).

But that's probably a pain when you have so many different devices to support.

(It also makes it harder to figure out what content people are watching... Though I think they use variable bitrate encoding which will fall to traffic analysis).

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#92

Post by jlgaddis » Mon, Sep 23, 2019, 5:13 AM UTC

yjftsjthsd-h wrote: post #85

loeg wrote: post #61

Earlier quoted context omitted.

Not all of them are landed in CURRENT yet, much less stable/12. They'll be in 13.0; I can't speak to any future stable 12.2. On the other hand, CURRENT is pretty solid. A lot of folks, Netflix included, just run CURRENT.

Wait, they run CURRENT in production? Is that... safe?

In this particular (Netflix) case, what's the worst that could happen? One of their OCAs panics and falls over... and all customer traffic gets shifted to another OCA in the same or another location?

Sure, they might run into some bug in -CURRENT, but what's the chance of a bunch of their OCAs all hitting it at the same time?

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#93

Post by bt848 » Mon, Sep 23, 2019, 5:23 AM UTC

loeg wrote: post #77

bt848 wrote: post #62

So why is it important to have a multisocket NUMA machine? Why not just save yourself a lot of hassle by having one socket? I know that the previous generation AMD machine had unavoidable NUMA but the new one doesn't.

This talk is about Zen+ Epyc, not Zen2 (which is where the non-cache memory gets uniform). I don't know if they have release quality Epyc 7003 (Zen2) samples available yet, and if they do, NFLX probably isn't allowed to publish benchmarks about them. There's almost certainly still some value in their existing NUMA work even on Zen2, as things like L1/L2/L3 cache have locality even if memory and PCIe does not. Pretty…

Skylake-X has 44 PCI 3.0 lanes, that's 352GT/s or about 345gbps application bandwidth. It's certainly more than enough to push 100gbps from disk to net. These guys are pushing 200gbps, but they're doing it with two CPUs, two sets of NVMe devices, and two NICs, and a bunch of hacks to make the operating system pretend all this stuff is not in the same box. It seems way more straight-forward to me if they had made it all be actually NOT in the same box!

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#94

Post by morning_gelato » Mon, Sep 23, 2019, 5:26 AM UTC

yjftsjthsd-h wrote: post #85

loeg wrote: post #61

Earlier quoted context omitted.

Not all of them are landed in CURRENT yet, much less stable/12. They'll be in 13.0; I can't speak to any future stable 12.2. On the other hand, CURRENT is pretty solid. A lot of folks, Netflix included, just run CURRENT.

Wait, they run CURRENT in production? Is that... safe?

There's at least one video online [1] that talks about Netflix's process for internal FreeBSD releases. They do 5 weeks of development and then 5 weeks of testing.

[1] https://www.youtube.com/watch?v=vcyQBup-Gto (about the 12 minute mark)

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#95

Post by jdsully » Mon, Sep 23, 2019, 5:50 AM UTC

shaklee3 wrote: post #89

jdsully wrote: post #71

The Linux network stack has been the bane of my existence trying to squeeze more performance out of KeyDB. I really hope it gets this kind of love in the future. On Linux there’s a spinlock in do soft_irq that blocks even in non-blocking IO.

Why not use DPDK? All of these problems go away, and people have reported hitting 1Tbps on a single node.

We have!

https://docs.keydb.dev/blog/2019/06/17/blog-post/

Still though I hate seeing needless waste. There’s no reason the active thread needs to block on a soft_irq when there are unused cores to process them.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#96

Post by yardstick » Mon, Sep 23, 2019, 6:00 AM UTC

mlyle wrote: post #84

toast0 wrote: post #79

Earlier quoted context omitted.

> Is it really a competitive advantage though? It totally is. These things get installed into ISP racks. ISPs have limited space and power for these things. One box pushing 200Gbps in the same space is twice as good (although, note they have 4x100Gbps NICs in this generation, there's room for more perf here). Reducing the number of nodes helps with management of the network as well.

It's funny to think that pushing 196gbps is possibly just as little as ~8000 simultaneous streams. Also a shame that it is TLS'd. The content already has its own DRM/encryption.

What’s the problem with TLS? The performance overhead of it is minimal these days.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#97

Post by drewg123 » Mon, Sep 23, 2019, 6:00 AM UTC

bluedino wrote: post #37

Nice to see AMD replacing Intel, they've gone with EPYC 7551 & 7502P, from 2x Intel “Skylake” / “Cascade Lake” Xeon edit: it looks like they hit 200GB/s with both Intel and AMD

Yes, we haven't gone with anything. These are proof of concept prototype systems. They serve real customer traffic in my testing, but are unicorns at this point.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#98

Post by drewg123 » Mon, Sep 23, 2019, 6:06 AM UTC

bt848 wrote: post #62

So why is it important to have a multisocket NUMA machine? Why not just save yourself a lot of hassle by having one socket? I know that the previous generation AMD machine had unavoidable NUMA but the new one doesn't.

I've run this particular AMD system in 3 ways: - Non-NUMA - 2 nodes per socket - 4 nodes per socket

The 4NPS gives the best performance, followed by 2NPS, followed by non-NUMA. This surprised me as well.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#99

Post by toast0 » Mon, Sep 23, 2019, 6:11 AM UTC

yardstick wrote: post #96

mlyle wrote: post #84

Earlier quoted context omitted.

It's funny to think that pushing 196gbps is possibly just as little as ~8000 simultaneous streams. Also a shame that it is TLS'd. The content already has its own DRM/encryption.

What’s the problem with TLS? The performance overhead of it is minimal these days.

It depends on the application; but for bulk transit of already encrypted (presumably by DRM) data, it is a significant overhead for little value.

You can go back and look at Netflix presentations from when they switched to TLS, and they were no longer able to saturate their network cards (and here, they're at 50% on network still)

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#100

Post by drewg123 » Mon, Sep 23, 2019, 6:20 AM UTC

shaklee3 wrote: post #88

floatboth wrote: post #56

Earlier quoted context omitted.

I know Chelsio has crypto directly on the NIC, but are dedicated crypto accelerator cards a thing and are they ever worth it? Why leave the CPU idle when the CPU itself is a good crypto accelerator (AES-NI, ARMv8 crypto)? AMD Ryzen has a built-in crypto "decelerator" — a FreeBSD driver was written for the crypto engine, but it's disabled by default because it made everything slower than AES-NI. (Though I guess it wou…

From what I recall, the chelsio cards only support a mode of encryption suitable for storage devices, and it's not something you'd use for streaming media.

No, not true. The Chelsio card support GCM and CBC crypto in lookaside (like QAT) using the ccr(4) OCF driver, inline ("NIC TLS") with out - of - tree patches, and TLS offload in TOE mode.

Now that ktls is upstream, we are looking at using the ccr crypto acceleration. We've already tested them in inline mode. TOE is not an option for us, since we do innovation in the TCP stack.