Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Linux Services: Before accepting connections

theojulienne.io

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Scaling Linux Services: Before accepting connections

#21

Post by jeffbee »

cranekam wrote: post #16
jeffbee wrote: post #7

A good advertisement for userspace networking.

Why? Because this stuff is in the kernel and thus harder to see? I don’t expect moving it to userspace to reduce overall complexity —- just move it elsewhere.

Because it’s in the kernel you aren’t exposed to all the knobs. There are hundreds of parameters controlling Linux tcp behavior. Even experts overlook some aspects. Hoisting this up into your application makes it visible. Why should there be a system parameter that limits the accept backlog of your server, silently? It makes no sense.

Re: Scaling Linux Services: Before accepting connections

#22

Post by marcosdumay »

fabian2k wrote: post #18

What are some very rough estimates on when it makes sense to look at these low-level network settings when scaling an application? I assume the default settings are good enough for moderate loads, but at which point does this stuff become a bottleneck? Are the default setting here reasonable for most cases, or is it more like something that you should tune even if you're not really pushing any limits?

Gerenaly those will start to hurt on the single digit thousands of connections per second (per process). I'd say that it's much more relevant that you start monitoring those logs when you reach single digit hundreds of connections per second than that you set a point to act. (100s of connections/second is a pretty normal "just got traction" value, so if you see a steady usage, monitor.)

Of course, YMMV. High latency networks reduce those numbers.

Anyway, I don't see why the numbers aren't 100 times larger by default, but there's probably a reason.

Re: Scaling Linux Services: Before accepting connections

#24

Post by Twirrim »

easytiger wrote: post #10
brendangregg wrote: post #5

Nice, although if you want to explore networking with ad hoc tracing tools, please try bpftrace[0]. Only use BCC once you need argparse and other python libraries. Here's my bpftrace SYN backlog tool from BPF Performance Tools (2019 book, tools are online[1]): # tcpsynbl.bt Attaching 4 probes... Tracing SYN backlog size. Ctrl-C to end. ^C @backlog[backlog limit]: histogram of backlog size @backlog[128]: [0] 2 |@@@@@@…

A lot of the world still have to use RHEL 6b etc and don't have these tools available

Just a general observation, if you're on RHEL6 you've got around 4 months left until End of Life. (I know, there are folks out there still running CentOS 4 and prior)

Re: Scaling Linux Services: Before accepting connections

#25

Post by stjohnswarts »

jeffbee wrote: post #21
cranekam wrote: post #16

Earlier quoted context omitted.

Why? Because this stuff is in the kernel and thus harder to see? I don’t expect moving it to userspace to reduce overall complexity —- just move it elsewhere.

Because it’s in the kernel you aren’t exposed to all the knobs. There are hundreds of parameters controlling Linux tcp behavior. Even experts overlook some aspects. Hoisting this up into your application makes it visible. Why should there be a system parameter that limits the accept backlog of your server, silently? It makes no sense.

Because the people who wrote it think it's better being a bit more inaccessible in the kernel so that regular users don't shoot themselves in the foot thinking they know better than the designers what the values should be. The people who know what they're doing will be able to set the parameters regardless of where they're hiding.

Re: Scaling Linux Services: Before accepting connections

#26

Post by shanemhansen »

VWWHFSfQ wrote: post #19
fabian2k wrote: post #18

What are some very rough estimates on when it makes sense to look at these low-level network settings when scaling an application? I assume the default settings are good enough for moderate loads, but at which point does this stuff become a bottleneck? Are the default setting here reasonable for most cases, or is it more like something that you should tune even if you're not really pushing any limits?

My NGINX webserver configuration on AWS behind an ALB is: /etc/sysctl.conf: net.core.wmem_max = 12582912 net.core.rmem_max = 12582912 net.ipv4.tcp_rmem = 10240 87380 12582912 net.ipv4.tcp_wmem = 10240 87380 12582912 fs.file-max = 1000000 net.ipv4.ip_local_port_range = 1024 65535 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_max_syn_backlog = 262144 net.ipv4.tcp_syncookies = 0 net.ipv4.tcp_fin_tim…

I think for ALB you'll see pooled connections (http or http2) so I would expect the number of TCP connections to stay pretty low. In http2 it could theoretically be as low as one.

Re: Scaling Linux Services: Before accepting connections

#27

Post by jeffbee »

stjohnswarts wrote: post #25
jeffbee wrote: post #21

Earlier quoted context omitted.

Because it’s in the kernel you aren’t exposed to all the knobs. There are hundreds of parameters controlling Linux tcp behavior. Even experts overlook some aspects. Hoisting this up into your application makes it visible. Why should there be a system parameter that limits the accept backlog of your server, silently? It makes no sense.

Because the people who wrote it think it's better being a bit more inaccessible in the kernel so that regular users don't shoot themselves in the foot thinking they know better than the designers what the values should be. The people who know what they're doing will be able to set the parameters regardless of where they're hiding.

This condescending attitude is itself a strong argument against whatever it is you work on.

Re: Scaling Linux Services: Before accepting connections

#28

Post by cutemonster »

eximius wrote: post #13

Why do these values default so low? Seems like you could have a few hundred or 1k instead of 128 with relatively little memory overhead.

I've wondered about that me too. What if it's because in the 1990s there wasn't that much memory

Re: Scaling Linux Services: Before accepting connections

#29

Post by easytiger »

Twirrim wrote: post #24
easytiger wrote: post #10

Earlier quoted context omitted.

A lot of the world still have to use RHEL 6b etc and don't have these tools available

Just a general observation, if you're on RHEL6 you've got around 4 months left until End of Life. (I know, there are folks out there still running CentOS 4 and prior)

This is not quite accurate. Large institutions with very slow processes and onerous governance will be very much tied to RHEL 6 for some years. It indeed is a very important part of Redhat's business model. Enterprises will purchase extended support for RHEL 6 going up to 2024

Re: Scaling Linux Services: Before accepting connections

#30

Post by Twirrim »

easytiger wrote: post #29
Twirrim wrote: post #24

Earlier quoted context omitted.

Just a general observation, if you're on RHEL6 you've got around 4 months left until End of Life. (I know, there are folks out there still running CentOS 4 and prior)

This is not quite accurate. Large institutions with very slow processes and onerous governance will be very much tied to RHEL 6 for some years. It indeed is a very important part of Redhat's business model. Enterprises will purchase extended support for RHEL 6 going up to 2024

That extended support isn't as comprehensive as the standard support, though. With each stage the amount of components and the degree and types of patching etc reduces.
Post reply on HN