Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The C10M problem

c10m.robertgraham.com

Post reply on HN
Print view View original

21–30 of 116 posts

Re: The C10M problem

#21

Post by throwaway092834 »

wpietri wrote: post #13

On the one hand, I love this. There's an old-school, down-to-the-metal, efficiency-is-everything angle that resonates deeply with me. On the other hand, I worry that just means I'm old. There are a lot of perfectly competent developers out there that have very little idea about the concerns that motivate thinking like this C10M manifesto. I sometimes wonder if my urge toward efficiency something like my grandmother's…

There's plenty of room for both. Many useful, worthwhile applications won't ever really push a machine to its limits, rather they primarily involve bringing structure and order to mounds of complex business logic. And that's OK. Not everything needs to be optimized.

On the other side of the same token, many useful, worthwhile applications will absolutely depend on this level of optimization. There are some problems which simply can't be solved in a practical manner without it. And that's OK too.

Re: The C10M problem

#22

Post by stefan_kendall »

erichocean wrote: post #16

What's significant to me is that you can do this stuff today on stock Linux. No need to run weird single-purpose kernels, strange hypervisors, etc. You can SSH into your box. You can debug with gdb. Valgrind. Everything is normal...except the performance, which is just insane. Given how easy it is, there isn't really a good excuse anymore to not write data plane applications the "right" way, instead of jamming everyt…

Not really. Last I checked, stock Ubuntu was choking around 60k concurrent connections for no reason, and Fedora could handle a lot more. This was a couple years back, but I'd demand numbers before assuming the situation has changed.

Re: The C10M problem

#23

Post by EdwardDiego »

dxhdr wrote: post #20
EdwardDiego wrote: post #18

At the risk of sounding dumb, aren't we still limited to 65,534 ports on an interface?

Port numbers must only be unique for ip:port pairs. A TCP connection is identified by the "quadruple" source_ip:source_port, dest_ip:dest_port. You can have as many connections as you want on the same source_ip on port 80 as long as there aren't 65,535 to the same dest_ip (ie as long as the quadruple is unique).

Cheers for the info! I guess that's where I got the wrong idea from - attempting to stress test one machine from another machine, I'd always hit that limit, but now I understand why.

Re: The C10M problem

#24

Post by erichocean »

stefan_kendall wrote: post #22
erichocean wrote: post #16

What's significant to me is that you can do this stuff today on stock Linux. No need to run weird single-purpose kernels, strange hypervisors, etc. You can SSH into your box. You can debug with gdb. Valgrind. Everything is normal...except the performance, which is just insane. Given how easy it is, there isn't really a good excuse anymore to not write data plane applications the "right" way, instead of jamming everyt…

Not really. Last I checked, stock Ubuntu was choking around 60k concurrent connections for no reason, and Fedora could handle a lot more. This was a couple years back, but I'd demand numbers before assuming the situation has changed.

The entire purpose is to NOT route stuff through the kernel.

TFA explains how to do it, and I've done it myself. You can set a flag on the Linux kernel when it boots limiting it to the first N cores. I usually use 2. The remaining cores are completely idle—Linux will not schedule any threads on those cores.

Then you build an app that works more-or-less like Snabb Switch, which talks directly to the Ethernet adaptor, bi-passing the kernel (Ubuntu, Fedora, etc. isn't relevant in the least).

So, you launch your app as a normal userland app. For each of your app's threads, schedule them on the remaining CPU cores however you want (I schedule one thread per core). Linux will not schedule its own threads or threads from any other process on those cores, so you own them completely—it'll never context switch to another thread.

That means when you SSH in, it's running on a thread on cores 1 or 2 only. Same with every other Linux process but your own. Other than sucking up available memory bandwidth and potentially trashing your L2/L3 cache, these other processes don't impact your own app at all.

Thus, even though you're running stock Linux, and SSH and gdb works, and you've got a normal userland app, your app is the ONLY app running on the remaining cores, and you're talking directly to the hardware. It's just as fast as doing everything without a kernel, except it cost you 2 cores. IMO, it's more than worth it for the convenience.

This approach is so easy that there's really no reason not to do it. There are so many situations in the past where I wanted the performance of those single-app kernels, but it just wasn't worth the dev effort. That's no longer true.

Re: The C10M problem

#25

Post by electrograv »

wpietri wrote: post #13

On the one hand, I love this. There's an old-school, down-to-the-metal, efficiency-is-everything angle that resonates deeply with me. On the other hand, I worry that just means I'm old. There are a lot of perfectly competent developers out there that have very little idea about the concerns that motivate thinking like this C10M manifesto. I sometimes wonder if my urge toward efficiency something like my grandmother's…

It really just depends on how much of a computational surplus you have. Some tasks have become so incredibly easy relative to today's computational horsepower, that we can afford waste 90% of it if it means we can be 10% more productive.

But this isn't true everywhere. Imagine if Google's servers cost 100x more to run, if they didn't spend time to make their code efficient. Imagine if your video games ran at a frame rate 100x slower than they do now. Imagine if your phone sucked 100x more battery power, and lagged 100x longer after each tap.

Sounds like I'm making extreme hypothetical examples? Not really, when languages like Python and Ruby can be over 100x slower than other high level languages (e.g. Java, not to mention C/C++).

And of course, the counterpoint is usually "But most of the time is spent in the database/[some highly optimized library], not the high-level logic." To that I reply: You're absolutely right. Guess what those subsystems were written in? C/C++/Java, most likely.

If you're fortunate enough to have all your performance-critical components already highly optimized and written for you by experts at bare-to-the-metal efficiency, you probably don't have to think about this. But if you're working on a technology truly new and unique, chances are you're going to have to "get your hands dirty" at some point if you want to avoid paying 100x more than is necessary in server fees.

Re: The C10M problem

#26

Post by hueving »

stefan_kendall wrote: post #22
erichocean wrote: post #16

What's significant to me is that you can do this stuff today on stock Linux. No need to run weird single-purpose kernels, strange hypervisors, etc. You can SSH into your box. You can debug with gdb. Valgrind. Everything is normal...except the performance, which is just insane. Given how easy it is, there isn't really a good excuse anymore to not write data plane applications the "right" way, instead of jamming everyt…

Not really. Last I checked, stock Ubuntu was choking around 60k concurrent connections for no reason, and Fedora could handle a lot more. This was a couple years back, but I'd demand numbers before assuming the situation has changed.

Do you have any data about what the bottleneck was?

Re: The C10M problem

#27

Post by oscargrouch »

ubikation wrote: post #11

I think cheetah OS, the MIT exo kernel project proved this and halvm by Galois does pretty well for network speed that xen provides, but I forget by how much. The netmap freebsd/linux interface is awesome! I'm looking forward to seeing more examples of its use.

i would just love to see that exo kernel from MIT in practice some day in some OS.. i think the research is from the nineties, isnt?

Also, netmap from freebds was the first thing that come to my head, as a relief from the IO bottleneck from moderns systems..

As in the original C10k, freebsd to the rescue here.. since it was the first OS with the kqueue interface.. and now is netmap.. the numbers from the speedup in the original paper are astounding

Re: The C10M problem

#28

Post by kyrra »

wpietri wrote: post #13

On the one hand, I love this. There's an old-school, down-to-the-metal, efficiency-is-everything angle that resonates deeply with me. On the other hand, I worry that just means I'm old. There are a lot of perfectly competent developers out there that have very little idea about the concerns that motivate thinking like this C10M manifesto. I sometimes wonder if my urge toward efficiency something like my grandmother's…

Most of the time your time is better spent adding feature to an application then trying to tune performance in the ways he talks about on the site. Most of the time it is easier to scale across multiple machines and pay for the hardware then it is to put in the time to make a program this efficient.

But there are applications like Snort (which he seems to reference a lot) that can't be split onto multiple boxes easily, so tuning for max performance is very important and is a key feature of the application.

Re: The C10M problem

#30

Post by rdtsc »

erichocean wrote: post #24
stefan_kendall wrote: post #22

Earlier quoted context omitted.

Not really. Last I checked, stock Ubuntu was choking around 60k concurrent connections for no reason, and Fedora could handle a lot more. This was a couple years back, but I'd demand numbers before assuming the situation has changed.

The entire purpose is to NOT route stuff through the kernel. TFA explains how to do it, and I've done it myself. You can set a flag on the Linux kernel when it boots limiting it to the first N cores. I usually use 2. The remaining cores are completely idle—Linux will not schedule any threads on those cores. Then you build an app that works more-or-less like Snabb Switch, which talks directly to the Ethernet adaptor,…

Linux will often still schedule kernel threads to run on those cores so they are not totally isolated. Also cache effects. If your architecture shares caches between cores, sometimes it would be worth "wasting" a neighboring core to avoid ssh and gdb thrashing your cache.

Oh and also don't forget to set up IRQ affinity to avoid any of those cores to handle.

There is an interesting research done by Siemens, that takes this kind of isolation a step further and uses virtualization extensions to isolate resources (cores for example):

https://github.com/siemens/jailhouse

Post reply on HN