Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Surprising gRPC Client Bottleneck in Low-Latency Networks

blog.ydb.tech

Post reply on HN
Print view View original

11–20 of 22 posts

Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks

#11

Post by lacop »

littlecranky67 wrote: post #10
lacop wrote: post #8

Earlier quoted context omitted.

Doesn't initcwnd only apply as the initial value? I don't care that the first request on the gRPC channel is slow, but subsequent requests on the same channel reuse the TCP connection and should have larger window size. This works as long as the channel is actively being used, but after short inactivity (few hundred ms, unsure exactly) something appears to revert back.

Yes, in case of hot tcp connections congestion control should not be the issue.

Yeah that was my understanding too, hence I filed the bug (actually duplicate of older bug that was closed because poster didn't provide reproduction).

Still not sure if this is linux network configuration issue or grpc issue, but something is for sure broken if I can't send a ~1MB request and get response within roughly network RTT + server processing time.

Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks

#12

Post by ltbarcly3 »

gRPC is a very badly implemented system. I have gotten 25%-30%+ improvements in throughput just by monkeypatching client libraries for google cloud to force json api endpoint usage.

At least try something else besides gRPC when building systems so you have a baseline performance understanding. gRPC is OFTEN introducing performance breakdowns that goes unnoticed.

Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks

#13

Post by stock_toaster »

ltbarcly3 wrote: post #12

gRPC is a very badly implemented system. I have gotten 25%-30%+ improvements in throughput just by monkeypatching client libraries for google cloud to force json api endpoint usage. At least try something else besides gRPC when building systems so you have a baseline performance understanding. gRPC is OFTEN introducing performance breakdowns that goes unnoticed.

Have you done any comparisons with connect-rpc?

Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks

#14

Post by charleshn »

lacop wrote: post #11
littlecranky67 wrote: post #10

Earlier quoted context omitted.

Yes, in case of hot tcp connections congestion control should not be the issue.

Yeah that was my understanding too, hence I filed the bug (actually duplicate of older bug that was closed because poster didn't provide reproduction). Still not sure if this is linux network configuration issue or grpc issue, but something is for sure broken if I can't send a ~1MB request and get response within roughly network RTT + server processing time.

Could you check the value of your kernel's net.ipv4.tcp_slow_start_after_idle sysctl, and if it's non zero set it to 0?

Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks

#15

Post by atombender »

yuliyp wrote: post #2

If you have a single TCP connection, all the data flows through that connection, ultimately serializing at least some of the processing. Given that the workers are just responding with OK, no matter how many CPU cores you give to that you're still bound by the throughput of the IO thread (well by the minimum of the client and server IO thread). If you want more than 1 IO thread to share the load, you need more than o…

When I started out using the gRPC SDK in Go, I was really surprised to find that it created just one connection per client. You'd think someone had made a multiplexing client wrapper, but I haven't been able to find one, so inevitably I've just written lightweight pooling myself. (Actually, I did come across one, but the code was very low quality.)

Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks

#16

Post by littlecranky67 »

charleshn wrote: post #14
lacop wrote: post #11

Earlier quoted context omitted.

Yeah that was my understanding too, hence I filed the bug (actually duplicate of older bug that was closed because poster didn't provide reproduction). Still not sure if this is linux network configuration issue or grpc issue, but something is for sure broken if I can't send a ~1MB request and get response within roughly network RTT + server processing time.

Could you check the value of your kernel's net.ipv4.tcp_slow_start_after_idle sysctl, and if it's non zero set it to 0?

This sounds exactly like the culprit. I didn't knew there is a slow start after idle and it is set to 1 (active) by default.

I wonder if I should change this to 0 on my default desktop machines for all connections.

Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks

#17

Post by voxic11 »

otterley wrote: post #7
yuliyp wrote: post #4

Earlier quoted context omitted.

I don't think this is head-of-line blocking. That is, it's not like a single slow request causes starvation of other requests. The IO thread for the connection is grabbing and dispatching data to workers as fast as it can. All the requests are uniform, so it's not like one request would be bigger/harder to handle for that thread.

> First, we checked the number of TCP connections using lsof -i TCP:2137 and found that only a single TCP connection was used regardless of in-flight count. It's head-of-line blocking. When requests are serialized, the queue will grow as long as the time to service a request is longer than the interval between arriving requests. Queue growth is bad if sufficient capacity exists to service requests in parallel.

Requests are not serialized though. Http2/GRPC multiplexs multiple requests over one TCP connection.

Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks

#18

Post by lacop »

charleshn wrote: post #14
lacop wrote: post #11

Earlier quoted context omitted.

Yeah that was my understanding too, hence I filed the bug (actually duplicate of older bug that was closed because poster didn't provide reproduction). Still not sure if this is linux network configuration issue or grpc issue, but something is for sure broken if I can't send a ~1MB request and get response within roughly network RTT + server processing time.

Could you check the value of your kernel's net.ipv4.tcp_slow_start_after_idle sysctl, and if it's non zero set it to 0?

That seems to work, thank you!

Now latency is just RTT + server time + payloadsize/bandwidth, not multiple times RTT: https://github.com/grpc/grpc-go/issues/8436#issuecomment-311...

I was not aware of this setting, it's pretty unfortunate this is a system-level setting that can't be overridden on application layer, and the idle timeout can't be changed either. Will have to figure out how to safely make this change on the k8s service this is affecting...

Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks

#19

Post by yuliyp »

otterley wrote: post #7
yuliyp wrote: post #4

Earlier quoted context omitted.

I don't think this is head-of-line blocking. That is, it's not like a single slow request causes starvation of other requests. The IO thread for the connection is grabbing and dispatching data to workers as fast as it can. All the requests are uniform, so it's not like one request would be bigger/harder to handle for that thread.

> First, we checked the number of TCP connections using lsof -i TCP:2137 and found that only a single TCP connection was used regardless of in-flight count. It's head-of-line blocking. When requests are serialized, the queue will grow as long as the time to service a request is longer than the interval between arriving requests. Queue growth is bad if sufficient capacity exists to service requests in parallel.

I guess I'd thought of head-of-line-blocking as the delay from a slow request stalling subsequent ones beyond the throughput limits of the system: i.e. a slow-to-parse request causes other cheap requests to wait.

Re: The Surprising gRPC Client Bottleneck in Low-Latency Networks

#20

Post by otterley »

yuliyp wrote: post #19
otterley wrote: post #7

Earlier quoted context omitted.

> First, we checked the number of TCP connections using lsof -i TCP:2137 and found that only a single TCP connection was used regardless of in-flight count. It's head-of-line blocking. When requests are serialized, the queue will grow as long as the time to service a request is longer than the interval between arriving requests. Queue growth is bad if sufficient capacity exists to service requests in parallel.

I guess I'd thought of head-of-line-blocking as the delay from a slow request stalling subsequent ones beyond the throughput limits of the system: i.e. a slow-to-parse request causes other cheap requests to wait.

Everything in a queue is subject to head-of-line blocking by definition; it's really a question of whether the wait time meets your requirements.
Post reply on HN