The Surprising gRPC Client Bottleneck in Low-Latency Networks
1–10 of 22 posts
Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks
#2If you have a single TCP connection, all the data flows through that connection, ultimately serializing at least some of the processing. Given that the workers are just responding with OK, no matter how many CPU cores you give to that you're still bound by the throughput of the IO thread (well by the minimum of the client and server IO thread). If you want more than 1 IO thread to share the load, you need more than one TCP connection.
Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks
#3classic case of head of line blocking!
Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks
#4classic case of head of line blocking!
I don't think this is head-of-line blocking. That is, it's not like a single slow request causes starvation of other requests. The IO thread for the connection is grabbing and dispatching data to workers as fast as it can. All the requests are uniform, so it's not like one request would be bigger/harder to handle for that thread.
Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks
#5Somewhat related, I'm running into a gRPC latency issue in https://github.com/grpc/grpc-go/issues/8436
If request payload exceeds certain size the response latency goes from network RTT to double that, or triple.
Definitely something wrong with either TCP or HTTP/2 windowing as it doesn't send the full request without getting ACK from server first. But none of the gRPC windowing config options nor linux tcp_wmem/rmem settings work. Sending one byte request every few hundred milliseconds fixes it by keeping the gRPC channel / TCP connection active. Nagle / slow start is disabled.
Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks
#6Somewhat related, I'm running into a gRPC latency issue in https://github.com/grpc/grpc-go/issues/8436 If request payload exceeds certain size the response latency goes from network RTT to double that, or triple. Definitely something wrong with either TCP or HTTP/2 windowing as it doesn't send the full request without getting ACK from server first. But none of the gRPC windowing config options nor linux tcp_wmem/rmem…
sounds like classic tcp congestion window scaling delay. Sounds like your payload exceeds 10x initcwnd.
Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks
#7classic case of head of line blocking!
I don't think this is head-of-line blocking. That is, it's not like a single slow request causes starvation of other requests. The IO thread for the connection is grabbing and dispatching data to workers as fast as it can. All the requests are uniform, so it's not like one request would be bigger/harder to handle for that thread.
> First, we checked the number of TCP connections using lsof -i TCP:2137 and found that only a single TCP connection was used regardless of in-flight count.
It's head-of-line blocking. When requests are serialized, the queue will grow as long as the time to service a request is longer than the interval between arriving requests. Queue growth is bad if sufficient capacity exists to service requests in parallel.
Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks
#8Somewhat related, I'm running into a gRPC latency issue in https://github.com/grpc/grpc-go/issues/8436 If request payload exceeds certain size the response latency goes from network RTT to double that, or triple. Definitely something wrong with either TCP or HTTP/2 windowing as it doesn't send the full request without getting ACK from server first. But none of the gRPC windowing config options nor linux tcp_wmem/rmem…
sounds like classic tcp congestion window scaling delay. Sounds like your payload exceeds 10x initcwnd.
Doesn't initcwnd only apply as the initial value? I don't care that the first request on the gRPC channel is slow, but subsequent requests on the same channel reuse the TCP connection and should have larger window size. This works as long as the channel is actively being used, but after short inactivity (few hundred ms, unsure exactly) something appears to revert back.
Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks
#9Somewhat related, I'm running into a gRPC latency issue in https://github.com/grpc/grpc-go/issues/8436 If request payload exceeds certain size the response latency goes from network RTT to double that, or triple. Definitely something wrong with either TCP or HTTP/2 windowing as it doesn't send the full request without getting ACK from server first. But none of the gRPC windowing config options nor linux tcp_wmem/rmem…
That's indeed interesting, thank you for sharing.
Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks
#10Earlier quoted context omitted.
sounds like classic tcp congestion window scaling delay. Sounds like your payload exceeds 10x initcwnd.
Doesn't initcwnd only apply as the initial value? I don't care that the first request on the gRPC channel is slow, but subsequent requests on the same channel reuse the TCP connection and should have larger window size. This works as long as the channel is actively being used, but after short inactivity (few hundred ms, unsure exactly) something appears to revert back.
Yes, in case of hot tcp connections congestion control should not be the issue.