Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

trac.nginx.org

Post reply on HN
Print view View original

51–60 of 96 posts

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#51

Post by rbw »

SilasX wrote: post #36
marcosdumay wrote: post #31

Earlier quoted context omitted.

The first sequence ends with foo/bar existing. The second one ends with it not existing. I'd say that anybody that sends a pair of PUT/DELETE requests in fast succession over the web and expects a stable result is a fool. This should have no effect on practice, because nobody should be relying on the ordering anyway.

Ah, my mistake. I had always equated PUT with updating and assumed it should fail if it doesn't find the resource. Big oversight!

I think you're correct, actually. If you want to create a new object, it should be a POST. In a well designed RESTful service, a PUT on an object that doesn't exist should fail, and both of the PUT/DELETE orderings above should result in the same state of the world: the object does not exist.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#52

Post by fgonzag »

kodablah wrote: post #10
JensRantil wrote: post #8

This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this. Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this. Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout r…

"but that will also disable connection timeout retry which is not what you want!" Why is this not what you want? Are you using the reverse proxy as a load balancer to multiple servers? Otherwise, if it's 1:1 proxy (for something like SSL termination) wouldn't having nginx fail/timeout when the server does be acceptable?

Using NGINX as a reverse proxy is an extremely common scenario. In fact that's what I currently run (with a support subscription), but will be evaluating moving to HAProxy if their tech dept does not provide a way to resolve this issue (which is actually a very big deal for me, and I was not aware)

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#54

Post by mhotchen »

bungle wrote: post #50
mhotchen wrote: post #44

Earlier quoted context omitted.

We've hit 3 problems with nginx: 1. Exactly this, we had mystery double trades from our clients and it took us a long time to realise it was nginx assuming we timed out and routing traffic to the next server 2. It doesn't do health checks. When a server goes down it will send 1 out of every 8 real requests to the down server to see if it responds. Having disabled resubmitting of requests to avoid the double trade iss…

Nginx can also be used for other protocols, see stream block.

I didn't know that. Thanks for the correction.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#55

Post by skinowski »

mhotchen wrote: post #44
JensRantil wrote: post #8

This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this. Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this. Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout r…

We've hit 3 problems with nginx: 1. Exactly this, we had mystery double trades from our clients and it took us a long time to realise it was nginx assuming we timed out and routing traffic to the next server 2. It doesn't do health checks. When a server goes down it will send 1 out of every 8 real requests to the down server to see if it responds. Having disabled resubmitting of requests to avoid the double trade iss…

Regarding 3, buffering behavior is highly configurable in nginx.(eg. proxy_request_buffering, proxy_buffering on/off)

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#56

Post by bungle »

JensRantil wrote: post #8

This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this. Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this. Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout r…

Does this help: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#pro... http://nginx.org/en/docs/http/ngx_http_proxy_module.html#pro...

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#57

Post by mortenlarsen »

jimjag wrote: post #49
_hyn3 wrote: post #22

This behavior is non-compliant[1] with the RFC. Although (from the standpoint of the RFC), everything on the server side (including nginx itself) is considered the web application, nginx probably takes the implicit position that dealing with multiple requests on non-idempotent methods such as POST is really a problem that the proxied web app itself should cope with. But then nginx puts the web app in an untenable pos…

A pretty good overview of reverse proxies can be found in [1]... spoiler: nginx is NOT one of the best-of-breed. Compliance was/is indeed an issue. 1. http://www.slideshare.net/bryan_call/choosing-a-proxy-server...

Non-SlideShare link (I think it is the same): http://cdn.oreillystatic.com/en/assets/1/event/115/Choosing%...

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#58

Post by jessaustin »

colanderman wrote: post #40
xxs wrote: post #6

Earlier quoted context omitted.

There could be races between PUT/DELETE. There is no grantee how the retry was made.

This is why the If-Match and If-None-Match preconditions exist; they resolve PUT races by checking that the resource is in the expected state.

I think you mean If-Match and If-Unmodified-Since. Actually they are relevant for DELETE as well. E.g. you might not want to DELETE if another client has just PUT.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#60

Post by SilasX »

rbw wrote: post #51
SilasX wrote: post #36

Earlier quoted context omitted.

Ah, my mistake. I had always equated PUT with updating and assumed it should fail if it doesn't find the resource. Big oversight!

I think you're correct, actually. If you want to create a new object, it should be a POST. In a well designed RESTful service, a PUT on an object that doesn't exist should fail, and both of the PUT/DELETE orderings above should result in the same state of the world: the object does not exist.

I thought so too, but when I looked at the spec [1], it agreed with the others:

>>The PUT method requests that the enclosed entity be stored under the supplied Request-URI. If the Request-URI refers to an already existing resource, the enclosed entity SHOULD be considered as a modified version of the one residing on the origin server. If the Request-URI does not point to an existing resource, and that URI is capable of being defined as a new resource by the requesting user agent, the origin server can create the resource with that URI

[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html 9.6

Post reply on HN