The Surprising gRPC Client Bottleneck in Low-Latency Networks

Post by perfgeek » Sat, Jul 26, 2025, 4:00 AM UTC

lacop wrote: post #18

charleshn wrote: post #14

Earlier quoted context omitted.

Could you check the value of your kernel's net.ipv4.tcp_slow_start_after_idle sysctl, and if it's non zero set it to 0?

That seems to work, thank you! Now latency is just RTT + server time + payloadsize/bandwidth, not multiple times RTT: https://github.com/grpc/grpc-go/issues/8436#issuecomment-311... I was not aware of this setting, it's pretty unfortunate this is a system-level setting that can't be overridden on application layer, and the idle timeout can't be changed either. Will have to figure out how to safely make this change on…

As you can imagine, when a TCP connection is first established, it has no knowledge of the conditions on the network. Thus we have slow start. At the same time, when a TCP connection goes idle, it's information about the conditions on the network become increasingly stale. Thus we have slow start after idle. In the Linux stack at least, being idle longer than the RTT (perhaps the computed RTO) is interpreted as meaning the TCP connection's idea of network conditions is no longer valid.

An application won't know anything about background specifics of the network to which the system on which it is running is attached. A system administrator might. In that sense at least, it is reasonable that it is a system tunable rather than a per-connection setsockopt().

Post by otterley » Mon, Jul 28, 2025, 7:50 PM UTC

voxic11 wrote: post #17

otterley wrote: post #7

Earlier quoted context omitted.

> First, we checked the number of TCP connections using lsof -i TCP:2137 and found that only a single TCP connection was used regardless of in-flight count. It's head-of-line blocking. When requests are serialized, the queue will grow as long as the time to service a request is longer than the interval between arriving requests. Queue growth is bad if sufficient capacity exists to service requests in parallel.

Requests are not serialized though. Http2/GRPC multiplexs multiple requests over one TCP connection.

The TCP connection itself is a single bidirectional queue. So if packets are lost or delayed, then multiplexing doesn't buy you anything.

Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks

Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks