Scaling Linux Services: Before accepting connections

Post by brendangregg » Sat, Jul 04, 2020, 6:30 AM UTC

theojulienne wrote: post #9

brendangregg wrote: post #5

Nice, although if you want to explore networking with ad hoc tracing tools, please try bpftrace[0]. Only use BCC once you need argparse and other python libraries. Here's my bpftrace SYN backlog tool from BPF Performance Tools (2019 book, tools are online[1]): # tcpsynbl.bt Attaching 4 probes... Tracing SYN backlog size. Ctrl-C to end. ^C @backlog[backlog limit]: histogram of backlog size @backlog[128]: [0] 2 |@@@@@@…

Thanks for the suggestion! I did come across the `tcpsynbl.bt` script as I was writing up this post, but wanted to add the additional information around namespaces and report additional information, which didn't seem as trivial in `bpftrace` as it was in Python, but that might be my lack of familiarity with the DSL :)

Post by eximius » Sat, Jul 04, 2020, 7:34 AM UTC

Post by codys » Sat, Jul 04, 2020, 7:54 AM UTC

ampersandy wrote: post #12

Is there a particular reason the Linux kernel favors names like `somaxconn` instead of `socket_max_connections`? It seems like a rather straightforward improvement for readability; so, why are shorter, compressed names preferred?

Post by batter » Sat, Jul 04, 2020, 9:09 AM UTC

Post by VWWHFSfQ » Sat, Jul 04, 2020, 1:00 PM UTC

fabian2k wrote: post #18

What are some very rough estimates on when it makes sense to look at these low-level network settings when scaling an application? I assume the default settings are good enough for moderate loads, but at which point does this stuff become a bottleneck? Are the default setting here reasonable for most cases, or is it more like something that you should tune even if you're not really pushing any limits?

    net.core.wmem_max = 12582912
    net.core.rmem_max = 12582912
    net.ipv4.tcp_rmem = 10240 87380 12582912
    net.ipv4.tcp_wmem = 10240 87380 12582912
    fs.file-max = 1000000
    net.ipv4.ip_local_port_range = 1024 65535
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_tw_reuse = 1
    net.ipv4.tcp_max_syn_backlog = 262144
    net.ipv4.tcp_syncookies = 0
    net.ipv4.tcp_fin_timeout = 3
    net.ipv4.tcp_syn_retries = 2
    net.ipv4.tcp_synack_retries = 2
    net.ipv4.tcp_no_metrics_save = 1
    net.ipv4.tcp_max_orphans = 262144
    net.core.somaxconn = 1000000

    worker_rlimit_nofile 102400;

    events {
      worker_connections 102400;
      multi_accept on;
    }

    http {
        server {
          listen 80 default_server reuseport backlog=102400;
          ...
        }    
    }