Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Kernel Tuning for C500k

blog.urbanairship.com

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Linux Kernel Tuning for C500k

#11

Post by evgen »

Some other tricks that were not touched upon in the article, but which may apply depending on the nature of your traffic:

1) If you have lots of short connections and you want to tune the amount of time that the kernel will keep half-closed connections around then you can play around with changing the values of net.ipv4.tcp_fin_timeout, net.ipv4.tcp_tw_reuse, net.ipv4.tcp_tw_recycle, and net.ipv4.tcp_max_tw_buckets.

2) If you have a modern NIC then you probably need to tweak the txqueuelen in your ifconfig options.

3) If you are get hits from a large number of random browsers then sometimes setting net.ipv4.tcp_no_metrics_save and net.ipv4.tcp_moderate_rcvbuf to turn off cacheing of flow metrics helps.

4) Increase net.core.somaxconn to increase your listen queue size.

5) If you have a local firewall like iptables in place make sure you increase net.ipv4.ip_conntrack_max, direct your high-traffic ports to the NOTRACK table, and play around with all of the various net.ipv4.netfilter.ip_conntrack_tcp_timeout_* settings.

Re: Linux Kernel Tuning for C500k

#12

Post by bnoordhuis »

evgen wrote: post #11

Some other tricks that were not touched upon in the article, but which may apply depending on the nature of your traffic: 1) If you have lots of short connections and you want to tune the amount of time that the kernel will keep half-closed connections around then you can play around with changing the values of net.ipv4.tcp_fin_timeout, net.ipv4.tcp_tw_reuse, net.ipv4.tcp_tw_recycle, and net.ipv4.tcp_max_tw_buckets.…

Good tips. The only thing I would recommend against is setting net.core.somaxconn too high - a too large backlog at a time when your server is already resource constrained might just push it over the brink.

Re: Linux Kernel Tuning for C500k

#13

Post by robotadam »

plainOldText wrote: post #3

I'm wondering what are the major side effects of this. Hmm?

A good question. Shrinking TCP buffer sizes can have a negative performance impact when sending large amounts of data; our use case was keeping track of a large number of mostly silent connections, and so we benefit from the smaller memory footprint.

Re: Linux Kernel Tuning for C500k

#15

Post by wmf »

metachris wrote: post #5
chrisbolt wrote: post #2

My, how things have changed since the C10k problem... http://www.kegel.com/c10k.html

The approaches stayed pretty much the same: 1. Serve many clients with each thread 2. Serve one client with each server thread 3. Build the server code into the kernel

It's really #1. #2 will barely get you to 10K and definitely not to 500K, while #3 is too brittle (although if you're feeling lazy, I noticed that Solaris has SSL and HTTP reverse proxies in the kernel).

Re: Linux Kernel Tuning for C500k

#16

Post by sophacles »

Just a technical note on the 64K myth section. My understanding is that TCP connections track by the tuple (remote_host, remote_port, local_host, local_port) so a single client can have 64k unique connections to each port on a remote machine.

If that is actually the case, the document gets its myth correction wrong (by a lot) :)

Can anyone clarify this?

Re: Linux Kernel Tuning for C500k

#17

Post by superjared »

sophacles wrote: post #16

Just a technical note on the 64K myth section. My understanding is that TCP connections track by the tuple (remote_host, remote_port, local_host, local_port) so a single client can have 64k unique connections to each port on a remote machine. If that is actually the case, the document gets its myth correction wrong (by a lot) :) Can anyone clarify this?

You are right. The part I didn't really make clear is that we only serve on the single external port. Were we to use multiple, then yes, we could have 64k * 64k per IP pair.

Re: Linux Kernel Tuning for C500k

#18

Post by ashish01 »

This is interesting stuff. I jumped into node.js programming a while ago and will like to run similar tests on node.js. Can anyone tell me how client side load of 500K long lived connections achieved ? Is there a standard set of programs to achieve this or some custom scripts.

Re: Linux Kernel Tuning for C500k

#19

Post by sophacles »

superjared wrote: post #17
sophacles wrote: post #16

Just a technical note on the 64K myth section. My understanding is that TCP connections track by the tuple (remote_host, remote_port, local_host, local_port) so a single client can have 64k unique connections to each port on a remote machine. If that is actually the case, the document gets its myth correction wrong (by a lot) :) Can anyone clarify this?

You are right. The part I didn't really make clear is that we only serve on the single external port. Were we to use multiple, then yes, we could have 64k * 64k per IP pair.

Ahh -- that makes sense. Thanks for clarifying.
Post reply on HN