Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

trac.nginx.org

Post reply on HN
Print view View original

61–70 of 96 posts

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#61

Post by lukeschlather »

joosters wrote: post #29
lukeschlather wrote: post #14

Earlier quoted context omitted.

Sure, but the point is you should design your POSTs to be idempotent as well.

Well, they are defined as non-idempotent, so there's no reason why you 'should' design this way. You can't make every request idempotent. It's best to design so that duplicate POSTs are handled sensibly (e.g. you don't make a user pay for the same product twice), but the response to the second POST is unlikely to be the same as the first one, so they aren't idempotent. More difficult cases are where an action could l…

There's really no such thing as perfectly idempotent operations. But it's an ideal to be emulated as much as possible. Even (or maybe especially) when something appears to be by definition not idempotent.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#62

Post by owengarrett »

This is unfortunate behavior on timeout, and we've shared a workaround solution available using maps. There's a configuration example in this Gist: https://gist.github.com/thresheek/2fa6479ffb7aca710493.

We're also going to prioritize a complete fix in the product, and encourage your comments and input on this ticket: https://trac.nginx.org/nginx/ticket/488

Disclaimer: I work @ NGINX. Thanks, Owen

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#63

Post by owengarrett »

fgonzag wrote: post #52
kodablah wrote: post #10

Earlier quoted context omitted.

"but that will also disable connection timeout retry which is not what you want!" Why is this not what you want? Are you using the reverse proxy as a load balancer to multiple servers? Otherwise, if it's 1:1 proxy (for something like SSL termination) wouldn't having nginx fail/timeout when the server does be acceptable?

Using NGINX as a reverse proxy is an extremely common scenario. In fact that's what I currently run (with a support subscription), but will be evaluating moving to HAProxy if their tech dept does not provide a way to resolve this issue (which is actually a very big deal for me, and I was not aware)

This is Owen from NGINX. We have a workaround for this behavior (https://gist.github.com/thresheek/2fa6479ffb7aca710493), and are tracking a separate new feature request. Please submit a support ticket or send me an email, owen@nginx.com.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#64

Post by colanderman »

jessaustin wrote: post #58
colanderman wrote: post #40

Earlier quoted context omitted.

This is why the If-Match and If-None-Match preconditions exist; they resolve PUT races by checking that the resource is in the expected state.

I think you mean If-Match and If-Unmodified-Since. Actually they are relevant for DELETE as well. E.g. you might not want to DELETE if another client has just PUT.

If-None-Match: * ensures that another client hasn't created the resource you are trying to create. It is equally important as If-Match for resolving race conditions.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#65

Post by JensRantil »

bungle wrote: post #56
JensRantil wrote: post #8

This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this. Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this. Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout r…

Does this help: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#pro... http://nginx.org/en/docs/http/ngx_http_proxy_module.html#pro...

No. Requests will still be retried.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#66

Post by JensRantil »

gshulegaard wrote: post #45
JensRantil wrote: post #8

This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this. Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this. Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout r…

What about `proxy_next_upstream off;`?

If I temporarily bring an upstream application down for upgrade I want nginx to retry the next upstream. This is a very common scenario when doing reverse proxying. Disabling next upstream breaks this.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#69

Post by typewriter_t »

I can't reproduce it. I have nginx proxy_pass'ing to two upstreams and configured with proxy_next_upstream timeout;

One of the upstreams is running iptables -A OUTPUT -p tcp --sport 8080 --tcp-flags PSH PSH -j DROP

CURLing the nginx location configured for proxy_pass'ing returns 504 GATEWAY_TIMEOUT on half of the requests, as expected.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#70

Post by timthorn »

mhotchen wrote: post #44
JensRantil wrote: post #8

This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this. Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this. Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout r…

We've hit 3 problems with nginx: 1. Exactly this, we had mystery double trades from our clients and it took us a long time to realise it was nginx assuming we timed out and routing traffic to the next server 2. It doesn't do health checks. When a server goes down it will send 1 out of every 8 real requests to the down server to see if it responds. Having disabled resubmitting of requests to avoid the double trade iss…

Whilst Nginx doesn't do healthchecks, they are available in Nginx Plus. I do appreciate that it is a charged for product, but it has a number of strong features over and above the OSS version and of course support (who are very responsive indeed).
Post reply on HN