Linux Kernel Tuning for C500k

About the suggested sysctl.conf settings: I think you'd also need to adjust net.core.rmem_max and net.core.wmem_max in order for the net.ipv4.tcp_rmem and net.ipv4.tcp_wmem settings to be effective.

Furthermore it couldn't hurt to increase net.core.netdev_max_backlog, which is the maximum number of packets queued on the input side, when the interface receives packets faster than kernel can process them.

Re: Linux Kernel Tuning for C500k

#5

Post by metachris » Wed, Sep 29, 2010, 9:09 PM UTC

chrisbolt wrote: post #2

My, how things have changed since the C10k problem... http://www.kegel.com/c10k.html

The approaches stayed pretty much the same:

1. Serve many clients with each thread

2. Serve one client with each server thread

3. Build the server code into the kernel

Re: Linux Kernel Tuning for C500k

#6

Post by superjared » Wed, Sep 29, 2010, 9:10 PM UTC

metachris wrote: post #4

Interesting post, thanks for sharing! About the suggested sysctl.conf settings: I think you'd also need to adjust net.core.rmem_max and net.core.wmem_max in order for the net.ipv4.tcp_rmem and net.ipv4.tcp_wmem settings to be effective. Furthermore it couldn't hurt to increase net.core.netdev_max_backlog , which is the maximum number of packets queued on the input side, when the interface receives packets faster than…

Regarding the `net.core` parameters. We do modify those, but my assumption (probably wrong) was that the `net.ipv4` changes would override the core configs. I'll take a look and update the post. Good point about `netdev_max_backlog`, I need to read up on that one too.

Thanks for the feedback!

Re: Linux Kernel Tuning for C500k

#7

Post by superjared » Wed, Sep 29, 2010, 9:11 PM UTC

chrisbolt wrote: post #2

My, how things have changed since the C10k problem... http://www.kegel.com/c10k.html

The C10k solutions are effectively the same as for C500k, those being epoll (Linux), kqueue (BSD), etc. Our Java NIO server utilizes epoll to handle C500k.

Re: Linux Kernel Tuning for C500k

#8

Post by metachris » Wed, Sep 29, 2010, 9:15 PM UTC

superjared wrote: post #6

metachris wrote: post #4

Interesting post, thanks for sharing! About the suggested sysctl.conf settings: I think you'd also need to adjust net.core.rmem_max and net.core.wmem_max in order for the net.ipv4.tcp_rmem and net.ipv4.tcp_wmem settings to be effective. Furthermore it couldn't hurt to increase net.core.netdev_max_backlog , which is the maximum number of packets queued on the input side, when the interface receives packets faster than…

Regarding the `net.core` parameters. We do modify those, but my assumption (probably wrong) was that the `net.ipv4` changes would override the core configs. I'll take a look and update the post. Good point about `netdev_max_backlog`, I need to read up on that one too. Thanks for the feedback!

I'm playing around with the exact same things -- maximum concurrent tcp sockets on Amazon EC2 large instances.

How many did you get running?

Re: Linux Kernel Tuning for C500k

#9

Post by robotadam » Wed, Sep 29, 2010, 9:20 PM UTC

metachris wrote: post #8

superjared wrote: post #6

Earlier quoted context omitted.

Regarding the `net.core` parameters. We do modify those, but my assumption (probably wrong) was that the `net.ipv4` changes would override the core configs. I'll take a look and update the post. Good point about `netdev_max_backlog`, I need to read up on that one too. Thanks for the feedback!

I'm playing around with the exact same things -- maximum concurrent tcp sockets on Amazon EC2 large instances. How many did you get running?

We got over 500k/instance: http://blog.urbanairship.com/blog/2010/08/24/c500k-in-action...

Re: Linux Kernel Tuning for C500k

#10

Post by superjared » Wed, Sep 29, 2010, 9:22 PM UTC

metachris wrote: post #8

superjared wrote: post #6

Earlier quoted context omitted.

Regarding the `net.core` parameters. We do modify those, but my assumption (probably wrong) was that the `net.ipv4` changes would override the core configs. I'll take a look and update the post. Good point about `netdev_max_backlog`, I need to read up on that one too. Thanks for the feedback!

I'm playing around with the exact same things -- maximum concurrent tcp sockets on Amazon EC2 large instances. How many did you get running?

On an EC2 Large we hit 520K.