111–120 of 137 posts
Earlier quoted context omitted.
Why not use DPDK? All of these problems go away, and people have reported hitting 1Tbps on a single node.
We have! https://docs.keydb.dev/blog/2019/06/17/blog-post/ Still though I hate seeing needless waste. There’s no reason the active thread needs to block on a soft_irq when there are unused cores to process them.
Earlier quoted context omitted.
I like it, I think its spreading, and I think its a good thing for the world. "Top U.S. CEOs say companies should put social responsibility above profit" https://www.reuters.com/article/us-jp-morgan-business-roundt...
The part that worries me is that companies are increasingly seeing it as their social responsibility to gang up and destroy "bad" people like Stallman.
So why is it important to have a multisocket NUMA machine? Why not just save yourself a lot of hassle by having one socket? I know that the previous generation AMD machine had unavoidable NUMA but the new one doesn't.
Communication over DDR4 is way faster than communication over PCIe Ethernet. Even 40Gbit is slow compared to RAM.
Earlier quoted context omitted.
What made Netflix pick FreeBSD over Linux in the first place?
BSDs unwavering commitment to reliability. A Linux kernel crashes. A lot. FreeBSD ran cdrom.com for 365+ days without rebooting. Just saying
Earlier quoted context omitted.
This talk is about Zen+ Epyc, not Zen2 (which is where the non-cache memory gets uniform). I don't know if they have release quality Epyc 7003 (Zen2) samples available yet, and if they do, NFLX probably isn't allowed to publish benchmarks about them. There's almost certainly still some value in their existing NUMA work even on Zen2, as things like L1/L2/L3 cache have locality even if memory and PCIe does not. Pretty…
Skylake-X has 44 PCI 3.0 lanes, that's 352GT/s or about 345gbps application bandwidth. It's certainly more than enough to push 100gbps from disk to net. These guys are pushing 200gbps, but they're doing it with two CPUs, two sets of NVMe devices, and two NICs, and a bunch of hacks to make the operating system pretend all this stuff is not in the same box. It seems way more straight-forward to me if they had made it a…
We're in total agreement :-). Their dataflow model requires something like 2x that in PCIe bandwidth and 4x in memory in the optimal case, as covered in the slides. 2x200 gbps = 400 gbps, which is a bit more than 345 gbps.
Maybe they could push 345/2 = 172 Gbps out of a single Skylake-X, best case. For some workloads, that might be the right local optima! They must have decided that the marginal cost of a 2P system was worth the extra ~25 Gbps to saturate the 200 Gbps pipe fully.
> they're doing it with two CPUs, two sets of NVMe devices, and two NICs, and a bunch of hacks to make the operating system pretend all this stuff is not in the same box. It seems way more straight-forward to me if they had made it all be actually NOT in the same box!
I've spoken with NFLX engineers in the past and my recollection is that in many installations, NFLX only get to install one box. (Or something like that. Might just be a cost thing.) So they need to make that one box fast.
I guess the other factor is the IP management overhead discussed in the slides. Two boxes necessitates the costly 2nd IP, as far as I know. It's hard to imagine the cost of an IP address dominating the marginal cost of a 2P socket system and 2nd Xeon, but I guess AWS is friggin expensive.
Earlier quoted context omitted.
Why not use DPDK? All of these problems go away, and people have reported hitting 1Tbps on a single node.
We have! https://docs.keydb.dev/blog/2019/06/17/blog-post/ Still though I hate seeing needless waste. There’s no reason the active thread needs to block on a soft_irq when there are unused cores to process them.
Earlier quoted context omitted.
Not all of them are landed in CURRENT yet, much less stable/12. They'll be in 13.0; I can't speak to any future stable 12.2. On the other hand, CURRENT is pretty solid. A lot of folks, Netflix included, just run CURRENT.
Wait, they run CURRENT in production? Is that... safe?
Earlier quoted context omitted.
Intel has a product line called QAT ("QuickAssisT"?) that does crypto acceleration, as well as compression. I don't know how performant it is. There are definitely several older crypto accelerators that were faster than CPUs of the time; I don't know if any of them (outside of QAT) is still relevant. The AMD Zen1 Crypto Co-Processor is indeed slower than AESNI; I think it's mostly used by stuff like SecureBoot, TPM,…
Supposedly QAT is built into the chipsets of Skylake and above now. I've never seen anyone try it, though.
The part I know of that is built into the CPU is a DMA engine called I/OAT; it just does DMA and maybe basic checksum and RAID transformations. It is sometimes confused with QAT (I've personally confused the two...):
https://www.intel.com/content/www/us/en/wireless-network/acc...
Earlier quoted context omitted.
Supposedly QAT is built into the chipsets of Skylake and above now. I've never seen anyone try it, though.
By chipset, you mean northbridge? Or the CPU? The part I know of that is built into the CPU is a DMA engine called I/OAT; it just does DMA and maybe basic checksum and RAID transformations. It is sometimes confused with QAT (I've personally confused the two...): https://www.intel.com/content/www/us/en/wireless-network/acc...
Earlier quoted context omitted.
Communication over DDR4 is way faster than communication over PCIe Ethernet. Even 40Gbit is slow compared to RAM.
Maybe but this post is about making the two sides of the computer NOT communicate.
But if CPU#1 wants to access a file that is on CPU#2 NVMe nodes, NUMA allows you to share those files across memory (and its a "local" file according to the OS), instead of over NFS or SMB.
--------
And yes, as much as we like to pretend that there's no communication and everything scales horizontally... in practice... people like sharing files between systems. NUMA allows for these files (and other resources: such as PCIe network cards or GPUs) to be shared between systems at the speed of DDR4 memory.