Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use the Linux kernel's TCP stack

blog.cloudflare.com

Post reply on HN
Print view View original

21–30 of 54 posts

Re: Why we use the Linux kernel's TCP stack

#21

Post by Annatar »

With this scale of attack the Linux kernel is not enough for us. We must work around it.

...Or you could just use an operating system substrate based on illumos which utilizes the FireEngine, like for instance SmartOS, instead of having to invent workarounds or use one's own TCP/IP stack implementations:

http://www.baylisa.org/library/slides/2005/august2005.pdf

https://sunaytripathi.wordpress.com/2010/03/25/solaris-10-ne...

this technology has been available for over ten years now; designed from the ground up to scale across multiple hardware threads for high performance, by the experts in the problem domain.

Re: Why we use the Linux kernel's TCP stack

#22

Post by Annatar »

peterwwillis wrote: post #5

Physical limitations of bandwidth on packets per second: https://www.cisco.com/c/en/us/about/security-center/network-... Why the Linux kernel has a hard time processing more than 1-2M packets per core per second, and patches/improvements for the kernel: https://lwn.net/Articles/629155/ CloudFlare's kernel bypass blog post: https://blog.cloudflare.com/kernel-bypass/ A paper from NTop on doing 10G line rate packet proc…

They are not designed to be a custom tailored highest-performance cure-all for the highest possible theoretical computing throughput.

I beg to differ vehemently, as the FireEngine TCP/IP stack in illumos was designed to be the highest possible performance cure-all for highest possible throughput. I've posted the links above in another entry. That GNU/Linux's TCP/IP stack is hitting the limits does not mean that nobody else is capable of designing a high performance TCP/IP stack, and indeed, I have been able to max out 1 Gbit connection running Solaris 10 on a measly DELL R910. If the network administrator hadn't come running to "turn it off! Turn the damn thing off!", I would have maxed out a trunked 40 Gbit link too. I sure taught that guy a lesson that day, never again have I heard a peep about "not ever seeing anyone being able to max out a 1 Gbit connection with a single machine".

Only Solaris / illumos' FireEngine makes it possible.

Re: Why we use the Linux kernel's TCP stack

#23

Post by quitspamming »

Annatar wrote: post #21

With this scale of attack the Linux kernel is not enough for us. We must work around it. ...Or you could just use an operating system substrate based on illumos which utilizes the FireEngine , like for instance SmartOS, instead of having to invent workarounds or use one's own TCP/IP stack implementations: http://www.baylisa.org/library/slides/2005/august2005.pdf https://sunaytripathi.wordpress.com/2010/03/25/solaris-…

Quit spamming about stupid SmartOS, you try to shoehorn it in to every topic. You're like a Mormon Missionary for SmartOS and it is super annoying.

Re: Why we use the Linux kernel's TCP stack

#24

Post by oddsignals »

quitspamming wrote: post #23
Annatar wrote: post #21

With this scale of attack the Linux kernel is not enough for us. We must work around it. ...Or you could just use an operating system substrate based on illumos which utilizes the FireEngine , like for instance SmartOS, instead of having to invent workarounds or use one's own TCP/IP stack implementations: http://www.baylisa.org/library/slides/2005/august2005.pdf https://sunaytripathi.wordpress.com/2010/03/25/solaris-…

Quit spamming about stupid SmartOS, you try to shoehorn it in to every topic. You're like a Mormon Missionary for SmartOS and it is super annoying.

I found his comment relevant and interesting enough, and judging by his posting history SmartOS is far from the only thing he comments about. It certainly added more to the discussion than yours did.

Re: Why we use the Linux kernel's TCP stack

#25

Post by jsnell »

Annatar wrote: post #22
peterwwillis wrote: post #5

Physical limitations of bandwidth on packets per second: https://www.cisco.com/c/en/us/about/security-center/network-... Why the Linux kernel has a hard time processing more than 1-2M packets per core per second, and patches/improvements for the kernel: https://lwn.net/Articles/629155/ CloudFlare's kernel bypass blog post: https://blog.cloudflare.com/kernel-bypass/ A paper from NTop on doing 10G line rate packet proc…

They are not designed to be a custom tailored highest-performance cure-all for the highest possible theoretical computing throughput. I beg to differ vehemently, as the FireEngine TCP/IP stack in illumos was designed to be the highest possible performance cure-all for highest possible throughput. I've posted the links above in another entry. That GNU/Linux's TCP/IP stack is hitting the limits does not mean that nobod…

Help me understand this. How is maxing out a 1Gbps connection with a 4U, 4 socket (so presumably 24-32 core) Xeon server supposed to be impressive?

Re: Why we use the Linux kernel's TCP stack

#27

Post by cyphar »

oddsignals wrote: post #24
quitspamming wrote: post #23

Earlier quoted context omitted.

Quit spamming about stupid SmartOS, you try to shoehorn it in to every topic. You're like a Mormon Missionary for SmartOS and it is super annoying.

I found his comment relevant and interesting enough, and judging by his posting history SmartOS is far from the only thing he comments about. It certainly added more to the discussion than yours did.

An incredibly large percentage of Annatar's posting history is a misunderstanding of something about GNU/Linux, followed by a pitch about SmartOS. It's not the only thing they talk a out, but it's the only posts that stick in my mind. While I find the history of free operating systems fascinating, it's quite dismissive to pretend that all possible problems that GNU/Linux faces today were solved "10+ years ago by experts in the problem domain".

Re: Why we use the Linux kernel's TCP stack

#28

Post by DblPlusUngood »

Annatar wrote: post #21

With this scale of attack the Linux kernel is not enough for us. We must work around it. ...Or you could just use an operating system substrate based on illumos which utilizes the FireEngine , like for instance SmartOS, instead of having to invent workarounds or use one's own TCP/IP stack implementations: http://www.baylisa.org/library/slides/2005/august2005.pdf https://sunaytripathi.wordpress.com/2010/03/25/solaris-…

It would be great if you could provide specific reasons why FireEngine is able to avoid the overheads which other projects avoid via kernel bypass.

Re: Why we use the Linux kernel's TCP stack

#29

Post by peterwwillis »

Annatar wrote: post #22
peterwwillis wrote: post #5

Physical limitations of bandwidth on packets per second: https://www.cisco.com/c/en/us/about/security-center/network-... Why the Linux kernel has a hard time processing more than 1-2M packets per core per second, and patches/improvements for the kernel: https://lwn.net/Articles/629155/ CloudFlare's kernel bypass blog post: https://blog.cloudflare.com/kernel-bypass/ A paper from NTop on doing 10G line rate packet proc…

They are not designed to be a custom tailored highest-performance cure-all for the highest possible theoretical computing throughput. I beg to differ vehemently, as the FireEngine TCP/IP stack in illumos was designed to be the highest possible performance cure-all for highest possible throughput. I've posted the links above in another entry. That GNU/Linux's TCP/IP stack is hitting the limits does not mean that nobod…

I didn't say anything was impossible. I said the Linux Kernel, and operating systems in general, have TCP/IP stacks not designed to facilitate the highest theoretical possible throughput.

And you mention "maxing out" a link. I bet you're talking about maximum throughput. That requires the maximum possible frame size. I, and the article, are talking about maximum frame rate, which requires the minimum possible frame size. The maximum rate results in about 18x more packets per second. (This is for a typical MTU of 1500; let's not get into jumbo frames...)

And you talk about 1Gbps. For a 10Gbps link, which is the most likely use case in the article, it results in 10x more packets per second, which requires a comparable scale in computing resources. But the stack is only designed to scale to the more likely use case of 1Gbps links; after 8 or so cores, performance can drop off precipitously.

So to hit the maximum packets per second of a 10Gbps link, you would need to handle 180x more packets per second than the max throughput of a 1Gbps link. Most stacks are not designed or performance tested with that in mind.

Re: Why we use the Linux kernel's TCP stack

#30

Post by peterwwillis »

jsnell wrote: post #19
gpderetta wrote: post #15

Earlier quoted context omitted.

Don't modern high performance network cards have multiple tx/rx queues which are virtualizable via IOMMU? That's a genuine question BTW, I've only a bit of experience with userspace networking with fully cooperating processes.

SR-IOV is good for actual virtualization, but it's pretty clumsy for trying to create isolation within a single VM. For example: - Every VF you create using SR-IOV will need to have a distinct MAC (and thus in practice different IP). But what you'd usually want for this use case is use the same IP for all apps, and do the split by destination port. - Another consequence of the previos point is that all apps would nee…

Use VPP for namespace-specific userland applications? https://wiki.fd.io/view/VPP/What_is_VPP%3F
Post reply on HN