Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Linux Services: Before accepting connections

theojulienne.io

Post reply on HN
Print view View original

1–10 of 30 posts

Re: Scaling Linux Services: Before accepting connections

#3

Post by IgorPartola »

This is well written. I never gave much thought to the resource usage during the period between SYN and accept. This article explained it very nicely. Also, now I’m curious why don’t these Linux limits don’t scale with the amount of RAM available? Like, yes on a low resource machine you wouldn’t want more than the default 128 for the backlog. But if I have 512GB of RAM then why not give me a backlog of a few thousand?

Re: Scaling Linux Services: Before accepting connections

#4

Post by yjftsjthsd-h »

IgorPartola wrote: post #3

This is well written. I never gave much thought to the resource usage during the period between SYN and accept. This article explained it very nicely. Also, now I’m curious why don’t these Linux limits don’t scale with the amount of RAM available? Like, yes on a low resource machine you wouldn’t want more than the default 128 for the backlog. But if I have 512GB of RAM then why not give me a backlog of a few thousand…

Obvious option: you might want that memory for processes. Dynamic scaling does seem sane, though.

Re: Scaling Linux Services: Before accepting connections

#5

Post by brendangregg »

Nice, although if you want to explore networking with ad hoc tracing tools, please try bpftrace[0]. Only use BCC once you need argparse and other python libraries.

Here's my bpftrace SYN backlog tool from BPF Performance Tools (2019 book, tools are online[1]):

  # tcpsynbl.bt
  Attaching 4 probes...
  Tracing SYN backlog size. Ctrl-C to end.
  ^C
  @backlog[backlog limit]: histogram of backlog size

  @backlog[128]: 
  [0]                    2 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|

  @backlog[500]: 
  [0]                 2783 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
  [1]                    9 |                                                    |
  [2, 4)                 4 |                                                    |
  [4, 8)                 1 |                                                    |
The source:

  #!/usr/local/bin/bpftrace
  
  #include 
  
  BEGIN
  {
          printf("Tracing SYN backlog size. Ctrl-C to end.\n");
  }
  
  kprobe:tcp_v4_syn_recv_sock,
  kprobe:tcp_v6_syn_recv_sock
  {
          $sock = (struct sock *)arg0;
          @backlog[$sock->sk_max_ack_backlog & 0xffffffff] =
              hist($sock->sk_ack_backlog);
          if ($sock->sk_ack_backlog > $sock->sk_max_ack_backlog) {
                  time("%H:%M:%S dropping a SYN.\n");
          }
  }
  
  END
  {
          printf("\n@backlog[backlog limit]: histogram of backlog size\n");
  }
This bpftrace tool is only 24 lines. The BCC tools in this post are >200 lines (and complex: needing to worry about bpf_probe_read() etc). The bpftrace version can also be easily modified to include extra details. I'm summarizing backlog length as a histogram since our prod hosts can accept thousands of connections per second.

[0] https://github.com/iovisor/bpftrace [1] https://github.com/brendangregg/bpf-perf-tools-book

Re: Scaling Linux Services: Before accepting connections

#6

Post by JoshTriplett »

IgorPartola wrote: post #3

This is well written. I never gave much thought to the resource usage during the period between SYN and accept. This article explained it very nicely. Also, now I’m curious why don’t these Linux limits don’t scale with the amount of RAM available? Like, yes on a low resource machine you wouldn’t want more than the default 128 for the backlog. But if I have 512GB of RAM then why not give me a backlog of a few thousand…

In general, Linux does favor automatic defaults over fixed static settings, if there's a reasonable heuristic to produce those defaults. But suppose, for instance, that you can't actually handle that many connections? There are two possibilities here: one is that you are processing connections fast enough to keep up, and the other is that you're not keeping up at all. In the former case, scaling the backlog up may help you keep up, though you may already have unacceptable latency. In the latter case, no amount of backlog will help you, and the backlog may make an attacker's job easier.

That said, there might well be a case for automatic backlog scaling. Or, for that matter, for increasing the default.

Re: Scaling Linux Services: Before accepting connections

#9

Post by theojulienne »

brendangregg wrote: post #5

Nice, although if you want to explore networking with ad hoc tracing tools, please try bpftrace[0]. Only use BCC once you need argparse and other python libraries. Here's my bpftrace SYN backlog tool from BPF Performance Tools (2019 book, tools are online[1]): # tcpsynbl.bt Attaching 4 probes... Tracing SYN backlog size. Ctrl-C to end. ^C @backlog[backlog limit]: histogram of backlog size @backlog[128]: [0] 2 |@@@@@@…

Thanks for the suggestion! I did come across the `tcpsynbl.bt` script as I was writing up this post, but wanted to add the additional information around namespaces and report additional information, which didn't seem as trivial in `bpftrace` as it was in Python, but that might be my lack of familiarity with the DSL :)

Re: Scaling Linux Services: Before accepting connections

#10

Post by easytiger »

brendangregg wrote: post #5

Nice, although if you want to explore networking with ad hoc tracing tools, please try bpftrace[0]. Only use BCC once you need argparse and other python libraries. Here's my bpftrace SYN backlog tool from BPF Performance Tools (2019 book, tools are online[1]): # tcpsynbl.bt Attaching 4 probes... Tracing SYN backlog size. Ctrl-C to end. ^C @backlog[backlog limit]: histogram of backlog size @backlog[128]: [0] 2 |@@@@@@…

A lot of the world still have to use RHEL 6b etc and don't have these tools available
Post reply on HN