Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use the Linux kernel's TCP stack

blog.cloudflare.com

Post reply on HN
Print view View original

31–40 of 54 posts

Re: Why we use the Linux kernel's TCP stack

#31

Post by scott_s »

felixgallo wrote: post #14
joosters wrote: post #11

Earlier quoted context omitted.

I can't see why pinning is such an obvious choice. The kernel's scheduler may decide that CPU 1 should be woken to handle some new packets, because CPU 2 is busy. If you've pinned the socket to CPU 2, you may be losing out. I get that there are trade-offs between the two modes: pinning can provide better cache usage, you can avoid some locks (but indirectly make the kernel do the work for you) and so on, but I don't…

The kernel scheduler has no idea which application thread handles which socket. You can formulate an application level plan and then enforce your will with socket and cpu pinning.

Correct, but the difficulty is if your application must share the machine with any other application - even short lived ones. That, I think, is what joosters was alluding to. If the assumption that your application is the only consumer of system resources is broken, then you may see pathological scheduling behavior.

Re: Why we use the Linux kernel's TCP stack

#32

Post by scott_s »

kieranelby wrote: post #26

The argument about not being able to run SSH on the server seemed a bit weak, surely just stick two NICs in there, one for management + one for user-space stuff?

I think that's less an argument, and more of an example. You are correct, that is certainly something one can do. But I think the related argument is that now your system configurations are more complicated; you have tied your hardware and software together, for example. For some that may not be possible, and even if it is, some may not want to give up the abstraction that the kernel provides.

Re: Why we use the Linux kernel's TCP stack

#33

Post by achamayou »

scott_s wrote: post #31
felixgallo wrote: post #14

Earlier quoted context omitted.

The kernel scheduler has no idea which application thread handles which socket. You can formulate an application level plan and then enforce your will with socket and cpu pinning.

Correct, but the difficulty is if your application must share the machine with any other application - even short lived ones. That, I think, is what joosters was alluding to. If the assumption that your application is the only consumer of system resources is broken, then you may see pathological scheduling behavior.

You can set isolcpus to earmark some cores for your application, and let the kernel manage the rest.

Re: Why we use the Linux kernel's TCP stack

#34

Post by jhallenworld »

raphaelj wrote: post #6

I implemented an highly-scalable user-space TCP stack as part of my Master's thesis [1], last year. One doesn't use an user-space network stack because the Linux's network stack is slow (it's fast), but because it doesn't scale correctly on an high number of CPUs (> 8 cores) [2]. This is because the kernel suffers from some lock contention when accessing the table containing the socket descriptors. An user-space stac…

Cavium allows you to run multiple instances of the Linux kernel- one pinned to each core of their np (you get to have a window of shared memory between them). It would be interesting to try this on Intel.

Re: Why we use the Linux kernel's TCP stack

#35

Post by Annatar »

quitspamming wrote: post #23
Annatar wrote: post #21

With this scale of attack the Linux kernel is not enough for us. We must work around it. ...Or you could just use an operating system substrate based on illumos which utilizes the FireEngine , like for instance SmartOS, instead of having to invent workarounds or use one's own TCP/IP stack implementations: http://www.baylisa.org/library/slides/2005/august2005.pdf https://sunaytripathi.wordpress.com/2010/03/25/solaris-…

Quit spamming about stupid SmartOS, you try to shoehorn it in to every topic. You're like a Mormon Missionary for SmartOS and it is super annoying.

No, I'm just sick and tired of Linux and want my favorite OS to finally hit the mainstream, so there would be some job opportunities. (Linux became popular the same way, for those of you with a short memory.) Now that Linux is finally hitting scale, people are getting busted by the shoddy programming, hence discussions about in- or out of kernel TCP/IP stack, which is preposterous, since the OS is supposed to provide an interface to the hardware. And I make no apologies for being a SmartOS advocate, just to set the record straight.

Re: Why we use the Linux kernel's TCP stack

#36

Post by Annatar »

cyphar wrote: post #27
oddsignals wrote: post #24

Earlier quoted context omitted.

I found his comment relevant and interesting enough, and judging by his posting history SmartOS is far from the only thing he comments about. It certainly added more to the discussion than yours did.

An incredibly large percentage of Annatar's posting history is a misunderstanding of something about GNU/Linux, followed by a pitch about SmartOS. It's not the only thing they talk a out, but it's the only posts that stick in my mind. While I find the history of free operating systems fascinating, it's quite dismissive to pretend that all possible problems that GNU/Linux faces today were solved "10+ years ago by expe…

Misunderstanding? I develop on Linux day in and day out. Care to qualify that assertion?

it's quite dismissive to pretend that all possible problems that GNU/Linux faces today were solved "10+ years ago by experts in the problem domain".

As one of the principal kernel engineers of the FireEngine, yeah I think Sunay is the expert in the problem domain, having invented parallel enqueuing or what he terms "fanout", and Radia Perlman, who I believe collaborated with him on it invented the spanning tree protocol. If that doesn't make them the subject matter experts in the TCP/IP stack domain, then I have nothing more to add. And yes, some or the problems GNU/Linux is hitting today have been solved on Solaris more than ten, others more than twenty years ago. Solaris had large enterprises as paying customers throughout the nineties of the past century, and those customers both demanded and paid huge sums of money to have these types of problems solved, so in some cases illumos has up to 25 years of a headstart, and by the time GNU/Linux catches up, illumos will already be ahead, as the development is not standing still and it has professional kernel engineers working on the code base.

Re: Why we use the Linux kernel's TCP stack

#37

Post by Annatar »

DblPlusUngood wrote: post #28
Annatar wrote: post #21

With this scale of attack the Linux kernel is not enough for us. We must work around it. ...Or you could just use an operating system substrate based on illumos which utilizes the FireEngine , like for instance SmartOS, instead of having to invent workarounds or use one's own TCP/IP stack implementations: http://www.baylisa.org/library/slides/2005/august2005.pdf https://sunaytripathi.wordpress.com/2010/03/25/solaris-…

It would be great if you could provide specific reasons why FireEngine is able to avoid the overheads which other projects avoid via kernel bypass.

The main reason being that the packets are put into queues bound to hardware threads (VCPU's), which the illumos kernel treats as processors. Sunay, the principal author of the FireEngine explains it in detail in the second link I cited. Long story short, on illumos based systems network performance scales linearly with available processors, and on systems where the NIC's PHY would be faster, a kernel tunable, ip_squeue_fanout (edittable via /etc/system), enables one to change the packet processing methodology. One of the techniques which enable the FireEngine to provide high network performance is eschewing mutex locks in favor of multiple parallel queues and parallel queue drainage. Should I also explain how mutex locks function, and why they are detrimental to performance as opposed to enqueueing?

Re: Why we use the Linux kernel's TCP stack

#38

Post by blastrat »

peterwwillis wrote: post #18
blastrat wrote: post #12

Earlier quoted context omitted.

| The real answer ... is that operating systems are designed to help users and programs. They are not designed to be a custom tailored highest-performance cure-all ... Using one tcp stack helps users and programs more... I don't use that definition of operating systems, I use the unix virtual machine because it's ubiquitous and provides its same large core set of defined functionalities across many scales and platfor…

There is no such thing. (Was this an attempt at a troll?)

there is no such thing as what? i didn't posit the existence of something that does not exist.

I disagreed with your POV because I can hardly imagine a scenario where I would recommend dropping an OS like Linux. This article and discussion is about understanding what a particular performance problem in linux is about, and (many eyes) perhaps people will suggest plausible solutions.

Your post is somewhat dismissive of the effort, to my ear, and it stems from your reductionist view of an OS as just some more software in addition to the software required for a project app.

Re: Why we use the Linux kernel's TCP stack

#39

Post by Annatar »

peterwwillis wrote: post #29
Annatar wrote: post #22

Earlier quoted context omitted.

They are not designed to be a custom tailored highest-performance cure-all for the highest possible theoretical computing throughput. I beg to differ vehemently, as the FireEngine TCP/IP stack in illumos was designed to be the highest possible performance cure-all for highest possible throughput. I've posted the links above in another entry. That GNU/Linux's TCP/IP stack is hitting the limits does not mean that nobod…

I didn't say anything was impossible. I said the Linux Kernel, and operating systems in general, have TCP/IP stacks not designed to facilitate the highest theoretical possible throughput. And you mention "maxing out" a link. I bet you're talking about maximum throughput. That requires the maximum possible frame size. I, and the article, are talking about maximum frame rate , which requires the minimum possible frame…

Solaris 10 can drive a 10Gb link at 7.3Gbps (limited by PCI-X bandwidth) using 2x2.2Ghz opteron CPUs utilized at less than 50%

https://sunaytripathi.wordpress.com/2010/03/25/solaris-10-ne..., page nine.

Now the PCIe is multiple times faster than the old PCI-X, and in addition to being several times faster then the old Opteron 939 and 940 series, modern intel based systems have 80 CPU's or more.

Re: Why we use the Linux kernel's TCP stack

#40

Post by Annatar »

jsnell wrote: post #25
Annatar wrote: post #22

Earlier quoted context omitted.

They are not designed to be a custom tailored highest-performance cure-all for the highest possible theoretical computing throughput. I beg to differ vehemently, as the FireEngine TCP/IP stack in illumos was designed to be the highest possible performance cure-all for highest possible throughput. I've posted the links above in another entry. That GNU/Linux's TCP/IP stack is hitting the limits does not mean that nobod…

Help me understand this. How is maxing out a 1Gbps connection with a 4U, 4 socket (so presumably 24-32 core) Xeon server supposed to be impressive?

Considering I did it seven years ago, I think it's impressive in that context.
Post reply on HN