Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NUMA Siloing in the FreeBSD Network Stack [pdf]

people.freebsd.org

Post reply on HN
Print view View original

121–130 of 137 posts

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#121

Post by jdsully »

shaklee3 wrote: post #116
jdsully wrote: post #95

Earlier quoted context omitted.

We have! https://docs.keydb.dev/blog/2019/06/17/blog-post/ Still though I hate seeing needless waste. There’s no reason the active thread needs to block on a soft_irq when there are unused cores to process them.

Can you move all softirqs into different cores? That's usually one of the dpdk tuning steps.

My reading of the source code is there is no way to prevent this. The spinlock is guarding the check which determines if there is work to do.

Its been a few months and I had intended to go back and try using a try_lock(). But I’m not normally a kernel dev.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#122

Post by jdsully »

throw0101a wrote: post #108
jdsully wrote: post #71

The Linux network stack has been the bane of my existence trying to squeeze more performance out of KeyDB. I really hope it gets this kind of love in the future. On Linux there’s a spinlock in do soft_irq that blocks even in non-blocking IO.

> I really hope it gets this kind of love in the future. Have you considered using FreeBSD? ;)

Unfortunately the vast majority of KeyDB users run Linux.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#123

Post by toast0 »

throw0101a wrote: post #106
toast0 wrote: post #79

Earlier quoted context omitted.

> Is it really a competitive advantage though? It totally is. These things get installed into ISP racks. ISPs have limited space and power for these things. One box pushing 200Gbps in the same space is twice as good (although, note they have 4x100Gbps NICs in this generation, there's room for more perf here). Reducing the number of nodes helps with management of the network as well.

Yes, but the parent was talking sharing the competitive advantage of the patch. But (a) what other streaming services even have CDN appliances, and (b) of the ones that do, which are using FreeBSD as a base? If none of Netflix's competitors will use this patch, then it makes no material difference whether they share it or not.

A) at least FB and Google, probably Amazon too, plus you know, lots of CDNs have CDN appliances.

B) as far as I'm aware, everyone else is running Linux, except maybe one of the CDNs, but I don't know if that CDN is actually running appliances.

However, everyone and their uncle are launching a streaming service in the next 18 months, some of those might get enough traffic to justify appliances, and they can take Netflix's work here to get a competitive box off the bat.

Also, at any point, one of the current big guys can say "hey, netflix gets 200gbps from one box, why do we have 4? Why don't we try FreeBSD on some of these"

CDN appliances are pretty isolated, so it doesn't matter as much if most of their proprietary stack isn't ported. Chances are, they've got someone on staff with FreeBSD experience, and FreeBSD experience ages better, because there's less churn.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#124

Post by m11r »

o-__-o wrote: post #67
robbyt wrote: post #15

Earlier quoted context omitted.

Realistically, Disney, or whomever competition, is not going to have enough technical expertise to deploy FreeBSD.

I trained the entire Disney offshore replacement team. They understand FreeBSD very well.

Out of curiosity, does this mean that they’re using (or have used) FreeBSD in their infrastructure or for some other purpose, or just a high level of Unix competence in that team? Any interesting details that you’re able to share?

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#125

Post by xemdetia »

toast0 wrote: post #79
throw0101a wrote: post #64

Earlier quoted context omitted.

> ... just shares potential competitive advantages like this? Is it really a competitive advantage though? I would think that their brand and content library are the main ones. A CDN server doesn't really count IMHO: other streaming services can create their own appliances, though I'm guessing they'll use Linux due to its ubiquity. Though as a long-time FreeBSD fan and part-time user, I'm happy for the patches. I run…

> Is it really a competitive advantage though? It totally is. These things get installed into ISP racks. ISPs have limited space and power for these things. One box pushing 200Gbps in the same space is twice as good (although, note they have 4x100Gbps NICs in this generation, there's room for more perf here). Reducing the number of nodes helps with management of the network as well.

I would say that has less to do with Netflix and more the recent generation of Intel/EPYC server computing. These numbers seem pretty standard for this generation of Skylake/Cascade Lake and one NIC per NUMA also feels pretty standard to me. The patches also seem to be the usual problem when you are starting to care about NUMA, where there are a couple places where it just wasn't added but you need it.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#126

Post by mlyle »

im3w1l wrote: post #102
mlyle wrote: post #91

Earlier quoted context omitted.

Easy enough to make the content authenticate itself, instead of repeating cryptographic work for each viewer. E.g. every 64k block contains a SHA256 hash of the next 64k block. When a user seeks, you provide the hash of the first block they'll be receiving over a trusted channel. (It already can exist at start of stream RSA signed or whatever). But that's probably a pain when you have so many different devices to sup…

This exists in a more advanced form already https://wicg.github.io/webpackage/draft-yasskin-http-origin-...

No... it's not the same as signed HTTP exchanges, which you'd generally use inside TLS, and involves signing all of the data in an exchange. It wouldn't work very well for streaming, either.

The important feature of the scheme that I mentioned is that there's no per-client encryption needed, except maybe a small operation on each client seek in the stream. In general, you just serve the file, and the file contents are self-authenticating against tampering.

You might use signed HTTP exchanges to push the first hash in the chain on a seek, I guess. Or just run it over TLS. Or just have a 64k block pre-signed at the beginning stream with the hashes of 2k chosen seek points that the client can store.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#127

Post by [deleted] »

cperciva wrote: post #66
floatboth wrote: post #57

Earlier quoted context omitted.

Why is the hw.model on EPYC redacted, when the chips are listed in the slides? Secret engineering sample? :)

He said in the talk that yes, the chips used in these measurements were not publicly released. IIRC he said that they were slightly lower clock speed than the publicly available chips.

[deleted]

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#128

Post by aepiepaey »

continuations wrote: post #51
drewg123 wrote: post #25

Earlier quoted context omitted.

Author here.. It depends heavily on the workload. We have man years of optimizations for our CDN workload that we have been upstreaming to FreeBSD. See, for example, the other talk I co-presented regarding Kernel TLS on FreeBSD. https://people.freebsd.org/~gallatin/talks/euro2019-ktls.pdf

What made Netflix pick FreeBSD over Linux in the first place?

It was mentioned in the QnA section of a talk posted somewhere in this comment section (probably https://www.youtube.com/watch?v=vcyQBup-Gto).

At least originally, it was because of the license.

Re: NUMA Siloing in the FreeBSD Network Stack [pdf]

#130

Post by [deleted] »

shaklee3 wrote: post #119
loeg wrote: post #118

Earlier quoted context omitted.

By chipset, you mean northbridge? Or the CPU? The part I know of that is built into the CPU is a DMA engine called I/OAT; it just does DMA and maybe basic checksum and RAID transformations. It is sometimes confused with QAT (I've personally confused the two...): https://www.intel.com/content/www/us/en/wireless-network/acc...

The northbridge. my understanding is that they no longer sell the discrete cards to perform these tasks, and instead offload it to chips that come on the boards.

[deleted]
Post reply on HN