Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why we use the Linux kernel's TCP stack

blog.cloudflare.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: Why we use the Linux kernel's TCP stack

#41

Post by wmf »

Annatar wrote: post #39
peterwwillis wrote: post #29

Earlier quoted context omitted.

I didn't say anything was impossible. I said the Linux Kernel, and operating systems in general, have TCP/IP stacks not designed to facilitate the highest theoretical possible throughput. And you mention "maxing out" a link. I bet you're talking about maximum throughput. That requires the maximum possible frame size. I, and the article, are talking about maximum frame rate , which requires the minimum possible frame…

Solaris 10 can drive a 10Gb link at 7.3Gbps (limited by PCI-X bandwidth) using 2x2.2Ghz opteron CPUs utilized at less than 50% https://sunaytripathi.wordpress.com/2010/03/25/solaris-10-ne... , page nine. Now the PCIe is multiple times faster than the old PCI-X, and in addition to being several times faster then the old Opteron 939 and 940 series, modern intel based systems have 80 CPU's or more.

Let's call that 3.3 Gbps/GHz. Linux can now drive 25 Gbps with a single 4 GHz core, so that's over 6 Gbps/GHz. Given the number of years involved, it's hard to tell how much improvement is due to hardware and how much is software. But I'm not blown away.

Re: Why we use the Linux kernel's TCP stack

#42

Post by wmf »

Annatar wrote: post #37
DblPlusUngood wrote: post #28

Earlier quoted context omitted.

It would be great if you could provide specific reasons why FireEngine is able to avoid the overheads which other projects avoid via kernel bypass.

The main reason being that the packets are put into queues bound to hardware threads (VCPU's), which the illumos kernel treats as processors. Sunay, the principal author of the FireEngine explains it in detail in the second link I cited. Long story short, on illumos based systems network performance scales linearly with available processors, and on systems where the NIC's PHY would be faster, a kernel tunable, ip_squ…

This sounds pretty similar to RSS/RPS/RFS in Linux. Granted, that functionality was added more recently and may have even been inspired/copied from Solaris for all I know, but the past is a sunk cost. None of your "back in the day" comments convince me that Solaris today has any advantages over Linux today.

Re: Why we use the Linux kernel's TCP stack

#43

Post by rconti »

raphaelj wrote: post #6

I implemented an highly-scalable user-space TCP stack as part of my Master's thesis [1], last year. One doesn't use an user-space network stack because the Linux's network stack is slow (it's fast), but because it doesn't scale correctly on an high number of CPUs (> 8 cores) [2]. This is because the kernel suffers from some lock contention when accessing the table containing the socket descriptors. An user-space stac…

Tangential, but what are some good resources for understanding the limitations of Linux on larger systems? (8 socket, multi-TB, multi-10gig, NUMA, etc)? Over the years I've found that the trivial questions have saturated the internet and made it very hard to GoogleShoot complicated problems.

Re: Why we use the Linux kernel's TCP stack

#44

Post by peterwwillis »

blastrat wrote: post #38
peterwwillis wrote: post #18

Earlier quoted context omitted.

There is no such thing. (Was this an attempt at a troll?)

there is no such thing as what? i didn't posit the existence of something that does not exist. I disagreed with your POV because I can hardly imagine a scenario where I would recommend dropping an OS like Linux. This article and discussion is about understanding what a particular performance problem in linux is about, and (many eyes) perhaps people will suggest plausible solutions. Your post is somewhat dismissive of…

There is no such thing as a "unix virtual machine". Unless, perhaps, you include Solaris Containers, since Solaris is technically a Unix operating system. I don't know what other "virtual machines" you discuss, nor what features or implementation details you're generalizing about, but your main assertion is nonsense and the rest is fluff.

In terms of your assertion that the easier it is to design a highest-performance tcp/ip stack the more ubiquitous it will be, is also wrong. Cisco and other vendors all have userspace plug-in frameworks for stacks that reach the highest performance packet-analysis-per-core in the industry, and they sure as shit aren't ubiquitous.

An OS is basically fancy glue to help programs work together to make the user's life easier. This has always been the case, because people are pretty universally annoyed by having to feed 1000 custom punch cards to a mainframe every time they want to run a program.

Re: Why we use the Linux kernel's TCP stack

#45

Post by blastrat »

peterwwillis wrote: post #44
blastrat wrote: post #38

Earlier quoted context omitted.

there is no such thing as what? i didn't posit the existence of something that does not exist. I disagreed with your POV because I can hardly imagine a scenario where I would recommend dropping an OS like Linux. This article and discussion is about understanding what a particular performance problem in linux is about, and (many eyes) perhaps people will suggest plausible solutions. Your post is somewhat dismissive of…

There is no such thing as a "unix virtual machine". Unless, perhaps, you include Solaris Containers, since Solaris is technically a Unix operating system. I don't know what other "virtual machines" you discuss, nor what features or implementation details you're generalizing about, but your main assertion is nonsense and the rest is fluff. In terms of your assertion that the easier it is to design a highest-performanc…

I'm using a different (and I believe more accurate) definition of virtual machine, and I'm using it because I think it offers more insight.

There are many possible chips you might have running inside your workstation, and none of them are the hardware they pretend to be, they are many varieties of microcoded superscalars that emulate the functionality of an amd64 architecture. They are virtual amd64s. On top of those you run OS binaries tuned for your hardware that present an API consistent with Linux (or Windows, depending on the software layers you run). If you had a hardware implementation of the Unix/Linux API, then that would be a Linux machine, but otherwise you are running a Linux virtual machine.

It's the Linux physical machine that does not exist. Linux virtual machines abound. (and many of them run on processors other than amd64s)

EDIT: I'm not offered a reply link to my repliers, so I'll edit answer in here instead.

I use that definition of virtual machine because it's the one true definition; to use the "naive" definition is to be wrong. I thought I was making that apparent in my description. There is no actual hardware amd64, there are only microcoded emulators of the architecture. So right off the bat, you see that if we code in assembly language, we are coding to a virtual machine, not an actual machine.

And just as you can think of mathematics as nothing more than the manipulation of symbols, when we write software, we are arranging symbols to code a virtual machine; in assembly language; in C; in Haskell; etc. Many (most?) C language implementations have symbolic references to the operating system. But inside that black box, we know is actually a virtual implementation on top of another virtual implementation.

An analogy would be, if you learn to drive an automobile, you learn to drive all of them because they use the same arrangement (more or less) of controls. They are different physical hardware manifestations (some gas, some electric, some diesel) of the same virtual machine (steering wheel, go pedal, stop pedal).

It was taught to me as a more useful definition, and I embraced it. I think using virtual machine to refer only to products from VMware is more problematic definitionally.

And in particular I used it in this thread to respond to the post at the very top that seemed to suggest that all software is just software as if we can just discard the operating system when it doesn't do what we want.

This is all what they teach at MIT in the computer science curriculum, BTW, nothing weird or cultish, or DOWNVOTEY about it.

Re: Why we use the Linux kernel's TCP stack

#46

Post by anarazel »

blastrat wrote: post #45
peterwwillis wrote: post #44

Earlier quoted context omitted.

There is no such thing as a "unix virtual machine". Unless, perhaps, you include Solaris Containers, since Solaris is technically a Unix operating system. I don't know what other "virtual machines" you discuss, nor what features or implementation details you're generalizing about, but your main assertion is nonsense and the rest is fluff. In terms of your assertion that the easier it is to design a highest-performanc…

I'm using a different (and I believe more accurate) definition of virtual machine, and I'm using it because I think it offers more insight. There are many possible chips you might have running inside your workstation, and none of them are the hardware they pretend to be, they are many varieties of microcoded superscalars that emulate the functionality of an amd64 architecture. They are virtual amd64s. On top of those…

What's the point of that definition?

Re: Why we use the Linux kernel's TCP stack

#48

Post by blastrat »

anarazel wrote: post #46
blastrat wrote: post #45

Earlier quoted context omitted.

I'm using a different (and I believe more accurate) definition of virtual machine, and I'm using it because I think it offers more insight. There are many possible chips you might have running inside your workstation, and none of them are the hardware they pretend to be, they are many varieties of microcoded superscalars that emulate the functionality of an amd64 architecture. They are virtual amd64s. On top of those…

What's the point of that definition?

i replied above because reply here wasn't an option after i got downvoted

Re: Why we use the Linux kernel's TCP stack

#49

Post by cyphar »

Annatar wrote: post #36
cyphar wrote: post #27

Earlier quoted context omitted.

An incredibly large percentage of Annatar's posting history is a misunderstanding of something about GNU/Linux, followed by a pitch about SmartOS. It's not the only thing they talk a out, but it's the only posts that stick in my mind. While I find the history of free operating systems fascinating, it's quite dismissive to pretend that all possible problems that GNU/Linux faces today were solved "10+ years ago by expe…

Misunderstanding? I develop on Linux day in and day out. Care to qualify that assertion? it's quite dismissive to pretend that all possible problems that GNU/Linux faces today were solved "10+ years ago by experts in the problem domain". As one of the principal kernel engineers of the FireEngine, yeah I think Sunay is the expert in the problem domain, having invented parallel enqueuing or what he terms "fanout", and…

> Care to qualify that assertion?

The most recent example I can think of is you posting about containers on GNU/Linux[1], claiming that they were implemented primarily using cgroups (and that the main purpose was resource restrictions). That is not true, and hasn't been true for a long time (if ever). Yes, the very first upstream "container" primitive was cgroups -- but that was very quickly replaced with namespaces and cgroups took on the resource restriction role. What most people call "containers" was always about virtualization (ie isolation), and the isolation primitive in the Linux kernel is namespaces.

There are almost certainly more examples, but I don't feel like going through any more of your comment history at the moment.

> And yes, some or the problems GNU/Linux is hitting today have been solved on Solaris more than ten, others more than twenty years ago.

Believe it or not, but constraints have changed in the past 20 years. I'm not saying that illumos doesn't have awesome technology (it does), but it is not a panacea. I get it, you're an advocate for alternative free operating systems. Good for you. Solaris does have a 25 year headstart -- on solving problems 25 years old. Modern computing has many more problems that weren't even concieved 25 years ago (cloud and distributed computing being the main ones, as well as embedded devices which is something that Solaris can't put a candle to GNU/Linux on). So it's very dismissive to claim that Solaris has solved all problems that may face GNU/Linux. Both operating systems have problems they need to fix.

> and it has professional kernel engineers working on the code base

So does Linux, I'm missing your point here.

[1] https://news.ycombinator.com/item?id=11944847

Re: Why we use the Linux kernel's TCP stack

#50

Post by peterwwillis »

Annatar wrote: post #39
peterwwillis wrote: post #29

Earlier quoted context omitted.

I didn't say anything was impossible. I said the Linux Kernel, and operating systems in general, have TCP/IP stacks not designed to facilitate the highest theoretical possible throughput. And you mention "maxing out" a link. I bet you're talking about maximum throughput. That requires the maximum possible frame size. I, and the article, are talking about maximum frame rate , which requires the minimum possible frame…

Solaris 10 can drive a 10Gb link at 7.3Gbps (limited by PCI-X bandwidth) using 2x2.2Ghz opteron CPUs utilized at less than 50% https://sunaytripathi.wordpress.com/2010/03/25/solaris-10-ne... , page nine. Now the PCIe is multiple times faster than the old PCI-X, and in addition to being several times faster then the old Opteron 939 and 940 series, modern intel based systems have 80 CPU's or more.

You are still quoting throughput when i'm quoting frame/packet rate. On top of this the line you have in italics doesn't show up on that page (it shows up in two random websites that provide no detail as to that claim), I have no idea what "page nine" on my resolution monitor is compared to yours, and you're trying to suggest something about the PCI bus being a factor (which it isn't; the limiting factor is cycles per packet). So I don't think you understand what's going on.
Post reply on HN