101–110 of 112 posts
The client would in the middle of sending a new request, but the server would have already decided to close the connection and the request would fail.
I believe this is a common problem, can and yet the spec has nothing to address this obvious race condition.
Right?
too bad 99% of routers drop keepalives
TCP keepalives != HTTP keepalives.
I was burned hard by this in Azure. It seems that the default expiry time is around 4 minutes for the TCP load balancers. You can bump it to 30 min, but if I recall the default interval on Linux is 2 hours. Any long-standing idle TCP connections would get into a state where both sides believed they were connected, but the packets would get dropped to the floor. When the LB timed out, it didn't emit any FIN or RST packets, so neither side knew it had been torn down.
Fun debugging on that one. During the day there was enough activity to keep the connections alive, but at night they'd break. The overall behaviour was that the service worked great all day, but the first few actions out-of-business-hours would fail due to application-layer timeouts, and then everything would work great again until it had sat idle for a while.
The hidden danger, mentioned in the article, is if the client sends a second request while the server closes an idle connection. Until http/2, the client can't tell if the server closed the connection before or after it received the request. Many servers send a hint about the idle time out, but few client libraries process it (that I've seen). The larger the latency between server and client, the bigger deal this is.
This is only partially true. http/1.1 has well defined semantics for persistent connections. The server can send the header "Connection: Close" to indicate to the client it is closing the idle connection. All http/1.1 clients should respect that since it's in the RFC. The problem is many servers don't send this header when closing idle connections. nginx is a notorious example. But well behaving servers should be sen…
However, when the server holds the connection open for some amount of time and then decides to close it, it's not permitted for the server to send a response header, because there's no request to respond to. I would love to be wrong, but I don't think I am, because this scenario is mentioned in the RFC, "For example, a client might have started to send a new request at the same time that the server has decided to close the "idle" connection. From the server's point of view, the connection is being closed while it was idle, but from the client's point of view, a request is in progress." [1]
An example chain of events is:
t0 client opens connection (syn)
t1 server accepts connection (syn+ack)
t2 client sends first request
t3 server sends response and keeps connection open
...
t63 client sends second request
t63 (simultaneously within a margin of the one way trip time), server closes connection because it's been idle for 60 seconds
t64 client receives FIN
t64 server receives data on closed socket and sends RST
t65 client receives RST
http/2 improves this greatly because in this example, a compliant server will send goaway with last-stream-id 1 prior to closing the connection, and the client will know the second request was not processed and should be retried. It still suffers a latency penalty because it has to start a new connection, and it already wasted somewhere between a one way trip and a round trip.
[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8....
Earlier quoted context omitted.
You send heartbeats! There might be a max-connection-time but I haven't run into it, my connections being dropped through amazon infrastructure was solved by sending a few bytes (': ') every 5 seconds or so.
TCP keepalive should solve the problem too. Rather than HTTP keepalive. (i.e. To handle the case of "HTTP-Request", "huge delay", "final response". Rather than a streaming/chunking reply that is very long/slow.)
The hidden danger, mentioned in the article, is if the client sends a second request while the server closes an idle connection. Until http/2, the client can't tell if the server closed the connection before or after it received the request. Many servers send a hint about the idle time out, but few client libraries process it (that I've seen). The larger the latency between server and client, the bigger deal this is.
Why do you think so? The server should report a Status Code for your 2nd request. What header would carry this keepalive hint?
The Keep-Alive header [1] is optional, but has parameters timeout, indicating the idle timeout, and max, indicating the number of allowed requests. Max is useful for pipelining, to avoid sending requests that won't be processed; timeout is very helpful for avoiding sending requests when the server is about to close the socket.
[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Ke... The Connection response header is specificed to have two optional parameters, timeout, and max. timeout
Earlier quoted context omitted.
If it's the first request on a connection, and it appears that the server closes the connection, I have a reasonable expectation that the server doesn't care for my request. When the request times out, who knows -- most tcp stacks won't tell me it the server acked it, but many networks will fake acks these days anyway. On pipelined requests it's not too bad, you're not supposed to pipeline requests that aren't safe t…
“Reasonable” to a human, maybe. If you are trying to build a robust system, in which requests don’t get lost, the difference is in quantity but not in quality - you must robustly handle the uncertainty in both cases.
But, most people don't realizing the byzantine hell we all inhabit; and http(s) client library defaults for retrying apparently idempotent requests will often work well enough; but server idle configured less than client idle is much easier to trip over.
Earlier quoted context omitted.
Why do you think so? The server should report a Status Code for your 2nd request. What header would carry this keepalive hint?
I think this because I've seen the pcap traces. The server closed the connection before it received the 2nd request -- it can't possibly send a status code. See my timeline in a sibling comment. The Keep-Alive header [1] is optional, but has parameters timeout, indicating the idle timeout, and max, indicating the number of allowed requests. Max is useful for pipelining, to avoid sending requests that won't be process…
For the keep-alive header, you are right, I wasn't aware of it.
Or better yet, use gzip and inline all images as base64 encoded. The file size is very similar to raw data, and the number of requests with associated http headers is reduced.
By forcing base64, you're eliminating all the caching and using much more CPU power to parse that back into a binary image. You're also making the page load slower as the initial payload is bigger and image data has to be handled in line rather than asynchronously.
Earlier quoted context omitted.
Could you be more specific, though? What's more complicated? I'm legitimately curious because I know very little about HTTP 2, but at work (not a tiny startup) we recently enabled it and it turned out to be a trivial change. Unless you're implementing the networking layer of your backend yourself, it seems like a change with practically no cost or tradeoff, as long as your server software supports it.
I haven't implemented it myself, but here's some example scenarios: Policy: What is allowed architecturally and what isn't? Are there regulatory requirements? Do you have strict enforcement mechanisms? Instrumentation: Do you need to watch traffic going over the wire? Will your network filters flag it? Do you have application proxies that route traffic based on payload? How is it going to handle multiplexing if exist…
Earlier quoted context omitted.
I think this because I've seen the pcap traces. The server closed the connection before it received the 2nd request -- it can't possibly send a status code. See my timeline in a sibling comment. The Keep-Alive header [1] is optional, but has parameters timeout, indicating the idle timeout, and max, indicating the number of allowed requests. Max is useful for pipelining, to avoid sending requests that won't be process…
Regarding the first question, IMO the webserver should half-close the socket. So, the on the wire requests are rejected on TCP level, while the final response is being delivered. Of course, the client needs to deal with socket errors in addition to http status codes. For the keep-alive header, you are right, I wasn't aware of it.
In my scenario at time N, the connection is idle -- both sides have received all data the other has sent, all requests have received a response.
If the server half-closes (through shutdown) and sends a FIN, simultaneously with the client sending a new request; that enables the server to read the request, but not respond to it, so I don't see how that is helpful?
The problem from the client side is it's sent a request, and seemingly in response the socket is closed. That could indicate the server crashed on the request, or the server closed the socket because it was idle. If you have a request that you know or suspect shouldn't be made more than once, you shouldn't retry it on a new connection. Assuming the tcp packets from the server, you can actually take a good guess at causality, because the ACK number and TCP Timestamp indicate if the server saw your last transmission, but that information isn't exposed through normal the normal socket API; you could maybe guess based on round trip time too, but it is nicer in http/2 (or other protocols), where there is an explicit close message.