Earlier quoted context omitted.
Author here.. We're doing ~200Gb/s through the kernel.
That's why I carefully inserted my weasel words of "I think" ;) Thank you for your contributions. Great information.
Thanks! I often do the same :)
NUMA Siloing in the FreeBSD Network Stack [pdf]
31–40 of 137 posts
Re: NUMA Siloing in the FreeBSD Network Stack [pdf]
#32Isn't it great that we work in an industry where a company like Netflix, which has increasing competition lately, just shares potential competitive advantages like this? I get that, in part, this is a way to get people excited to work for Netflix, however, the people working on this are, probably, pretty proud of what they achieved and thus like to share it with us, their colleagues at other places of work and they h…
>just shares potential competitive advantages like this?
It is abit like Facebook Sharing their work on Servers Hardware and Database improvements. Ultimately they dont think any of those technology are their completive advantage , their core business is Social ( and Ad Tracking, but that is another topic ).
Similar to Netflix, and would be the same as Disney. Their core business is not in the delivery, but the content.
Having said that it is still very nice of them to share their work. FreeBSD, Ruby Rails, Chaos Engineering, there are lots of things I love about their Tech Stacks.
Re: NUMA Siloing in the FreeBSD Network Stack [pdf]
#33Earlier quoted context omitted.
> I get that, in part, this is a way to get people excited to work for Netflix It's even more amazing to me that this is a successful tactic in light of the fact that there's no shortage of people willing to work for ad-tech firms. If working for a company with good values was such a draw, you'd think it would give them more of a competitive advantage. Maybe the real reason is that Netflix's "good values" are genuine…
I don't see how netflix' "good values" are so much better than adtech. They're not exactly saving the planet, feeding the starving or growing artificial organs. Adtech is about getting eyeballs and converting that into money. Netflix is about getting eyeballs for hours and people paying for that privilege. Contributing to OSS projects they use is sort of the "not eating babies" baseline, even oracle and google do it!
Netflix doesn't involve stalking people online and then selling the data you derive to the highest bidder.
Re: NUMA Siloing in the FreeBSD Network Stack [pdf]
#34It’s my understanding that many organizations prefer *BSDs over Linux not because BSD is more performant ... but because you’re way more likely to get your organizations patch accepted into the upstream than Linux.
Re: NUMA Siloing in the FreeBSD Network Stack [pdf]
#35Good
Re: NUMA Siloing in the FreeBSD Network Stack [pdf]
#36I wonder how this would compare with Receive Side Scaling (RSS), which you can use to pin NIC queues to cpus, and then pin the rest of the handling to the same cpu, avoiding a significant amount of interprocess communication. NUMA concerns may be more important than CPU pining within a domain though.
RSS is more about spreading a workload over multiple cores to not overwhelm any single core, particularly in high PPS situations.
This is about PCI-E devices being hooked up to a NUMA node to avoid saturating the link between nodes. There's a limited amount of bandwidth, and crossing nodes saturates this and increases latency, both of which will have limiting effects on your total possible throughput.
RSS configuration doesn't need you to set up your hardware in any specific way, with this you need to ensure that the set of disks and nic are hooked up to coupled domains - e.g. if you place the two NICs on the same NUMA node, then no software configuration is going to fix that, and you'd have to go and physically rearrange things to fix it.
You might still use RSS to distributed the workload across multiple cores within that NUMA domain when using this setup.
Re: NUMA Siloing in the FreeBSD Network Stack [pdf]
#37Nice to see AMD replacing Intel, they've gone with EPYC 7551 & 7502P, from 2x Intel “Skylake” / “Cascade Lake” Xeon
edit: it looks like they hit 200GB/s with both Intel and AMD
Re: NUMA Siloing in the FreeBSD Network Stack [pdf]
#38Earlier quoted context omitted.
Guessing that they have managed services in aws for maintaining all the nodes and where they are. If each box has 2x the ips, they would need more mgmt resources to keep track of what is on that box, if it's online, stats, etc.
Yes, exactly. When we first started the project, I was told that multiple IPs were not an option for that reason.
Was that a technical or administrative restriction? I'm interested in how an organization makes a decision like that, and then what channels are used to reverse it.
Re: NUMA Siloing in the FreeBSD Network Stack [pdf]
#39Isn't it great that we work in an industry where a company like Netflix, which has increasing competition lately, just shares potential competitive advantages like this? I get that, in part, this is a way to get people excited to work for Netflix, however, the people working on this are, probably, pretty proud of what they achieved and thus like to share it with us, their colleagues at other places of work and they h…
Realistically, Disney, or whomever competition, is not going to have enough technical expertise to deploy FreeBSD.
Disney’s animation studio is based on Linux infrastructure: https://www.velvetjobs.com/job-posting/sr-systems-engineer-l...
Re: NUMA Siloing in the FreeBSD Network Stack [pdf]
#40This may be a dumb question, but why would streaming video need to be encrypted? Is this just part of the "encrypt everything" best practice these days? Is there metadata accompanying the video data that shouldn't be unencrypted? Is it just so it's not possible to eavesdrop on the fact that I'm watching "Marvelous Mrs. Maisel"?