Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default
1–10 of 96 posts
Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default
#2I've been bitten more than once by this.
Example situation, you have a request to process an uploaded file which is only for admin purpose so you didn't take the time to use a queuing system to do the heavy lifting in the background. Then, your customer uploads a file that takes much longer than normal and the request times out, that file is then sent multiple times to the app server and the user sees multiple uploads..
The behavior in term of errors should definitely be different if the sending request failed (in which case resending to the next upstream is fine) and if receiving the response failed (in which case it's often not a good idea to resend)
Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default
#3This is why idempotence is so important.
Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default
#4This almost seems non-compliant w/ the action HTTP spec.
Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default
#5This is why idempotence is so important.
But it's only the POST that this is a problem for, right? PUT and DELETE are supposed to be idempotent so retries are okay, yes?
Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default
#6Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default
#7This is why idempotence is so important.
But it's only the POST that this is a problem for, right? PUT and DELETE are supposed to be idempotent so retries are okay, yes?
Yes. Incorrectly implementing the HTTP spec is widespread though, I imagine especially with PUT.
People are writing custom HTTP application servers and making PUT do anything and everything. Idempotence doesn't usually enter the picture.
Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default
#8This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this.
Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this.
Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout retry which is not what you want!
Increasing `proxy_connect_timeout` is not an option, because then you risk filling up too many connections in the nginx instance if the upstream server swallows SYN packets or whatnot.
The real workaround: Use haproxy. Serioously.
Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default
#9Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default
#10This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this. Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this. Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout r…
"but that will also disable connection timeout retry which is not what you want!"
Why is this not what you want? Are you using the reverse proxy as a load balancer to multiple servers? Otherwise, if it's 1:1 proxy (for something like SSL termination) wouldn't having nginx fail/timeout when the server does be acceptable?