Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

trac.nginx.org

Post reply on HN
Print view View original

71–80 of 96 posts

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#71

Post by todd17 »

_hyn3 wrote: post #22

This behavior is non-compliant[1] with the RFC. Although (from the standpoint of the RFC), everything on the server side (including nginx itself) is considered the web application, nginx probably takes the implicit position that dealing with multiple requests on non-idempotent methods such as POST is really a problem that the proxied web app itself should cope with. But then nginx puts the web app in an untenable pos…

Scenario 1:

1. nginx times out while server processes request

2. nginx makes request to second server and second server returns "account already exists"

Scenario 2:

1. nginx times out while server processes request and returns error

2. user attempts to create account again and server returns "account already exists"

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#72

Post by kvz »

skinowski wrote: post #55
mhotchen wrote: post #44

Earlier quoted context omitted.

We've hit 3 problems with nginx: 1. Exactly this, we had mystery double trades from our clients and it took us a long time to realise it was nginx assuming we timed out and routing traffic to the next server 2. It doesn't do health checks. When a server goes down it will send 1 out of every 8 real requests to the down server to see if it responds. Having disabled resubmitting of requests to avoid the double trade iss…

Regarding 3, buffering behavior is highly configurable in nginx.(eg. proxy_request_buffering, proxy_buffering on/off)

It's only as of 1.8 that you can disable buffering of incoming requests though. Just a few month iirc.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#73

Post by xetorthio »

JensRantil wrote: post #43
marcosnils wrote: post #32

We had this same problem some time ago in our company. That's why we came up with this. https://github.com/xetorthio/nginx-upstream-idempotent

Cool. Did you ever consider patching nginx upstream instead?

Yes. Before implementing this module we contacted nginx developers and they didn't think it is a problem. This is why we had to create our own module.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#74

Post by thedufer »

rbw wrote: post #51
SilasX wrote: post #36

Earlier quoted context omitted.

Ah, my mistake. I had always equated PUT with updating and assumed it should fail if it doesn't find the resource. Big oversight!

I think you're correct, actually. If you want to create a new object, it should be a POST. In a well designed RESTful service, a PUT on an object that doesn't exist should fail, and both of the PUT/DELETE orderings above should result in the same state of the world: the object does not exist.

You're probably thinking of PATCH. In many, if not most, RESTful services, PUT is given PATCH semantics. PUT is supposed to be insert-or-update.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#75

Post by thedufer »

33degrees wrote: post #34
joosters wrote: post #29

Earlier quoted context omitted.

Well, they are defined as non-idempotent, so there's no reason why you 'should' design this way. You can't make every request idempotent. It's best to design so that duplicate POSTs are handled sensibly (e.g. you don't make a user pay for the same product twice), but the response to the second POST is unlikely to be the same as the first one, so they aren't idempotent. More difficult cases are where an action could l…

I think the most of these cases can be handled via PUT i.e. update a cart so it contains these items. That way you keep all the state on the client.

That's great unless people want to shop in multiple browser tabs or anything like that. The real solution here is to use the fact that we have POST which is specced as non-idempotent and not depend on the very small set of technologies that purposefully disobeys the spec.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#76

Post by orthecreedence »

skinowski wrote: post #55
mhotchen wrote: post #44

Earlier quoted context omitted.

We've hit 3 problems with nginx: 1. Exactly this, we had mystery double trades from our clients and it took us a long time to realise it was nginx assuming we timed out and routing traffic to the next server 2. It doesn't do health checks. When a server goes down it will send 1 out of every 8 real requests to the down server to see if it responds. Having disabled resubmitting of requests to avoid the double trade iss…

Regarding 3, buffering behavior is highly configurable in nginx.(eg. proxy_request_buffering, proxy_buffering on/off)

Wrote a post where I ran into (and fixed) this problem with streaming uploads through nginx: http://killtheradio.net/technology/nginx-returns-error-on-fi...

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#77

Post by coldtea »

colanderman wrote: post #35

How is one supposed to take seriously web infrastructure software that exhibits such a basic failure of understanding core web standards? From even a cursory reading of the HTTP RFCs one will understand that "POST = unsafe = don't retry after request sent = return 504 on reply timeout". I mean, a bug's a bug; but this was known for two years!

>How is one supposed to take seriously web infrastructure software that exhibits such a basic failure of understanding core web standards?

How? Probably based on the fact that otherwise it's a frigging great app that powers like 15% of the web, including some of the biggest sites out there.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#78

Post by fgonzag »

owengarrett wrote: post #63
fgonzag wrote: post #52

Earlier quoted context omitted.

Using NGINX as a reverse proxy is an extremely common scenario. In fact that's what I currently run (with a support subscription), but will be evaluating moving to HAProxy if their tech dept does not provide a way to resolve this issue (which is actually a very big deal for me, and I was not aware)

This is Owen from NGINX. We have a workaround for this behavior ( https://gist.github.com/thresheek/2fa6479ffb7aca710493 ), and are tracking a separate new feature request. Please submit a support ticket or send me an email, owen@nginx.com.

Thank you, I will be opening the ticket tomorrow. Regarding the gist you just posted, it seems this simply disables proxy_next_upstream for any and all non idempotent requests.

However what would really need to happen is to only disable proxy_next_upstream if data has been written or read from the backend(preferably configurable by backend or location for either of those two options). Right now you basically lose the redundancy in non-idempotent requests, and immediately return the error. Or maybe I read the configuration incorrectly.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#79

Post by jessaustin »

colanderman wrote: post #64
jessaustin wrote: post #58

Earlier quoted context omitted.

I think you mean If-Match and If-Unmodified-Since. Actually they are relevant for DELETE as well. E.g. you might not want to DELETE if another client has just PUT.

If-None-Match: * ensures that another client hasn't created the resource you are trying to create. It is equally important as If-Match for resolving race conditions.

It's fairly uncommon to use PUT for resource creation. In that case, however, if the server supported it, yes you could use If-None-Match. I really have to wonder about the architecture of the system, however, if two clients can simultaneously decide to create the same resource rather than two similar resources.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#80

Post by JensRantil »

owengarrett wrote: post #62

This is unfortunate behavior on timeout, and we've shared a workaround solution available using maps. There's a configuration example in this Gist: https://gist.github.com/thresheek/2fa6479ffb7aca710493 . We're also going to prioritize a complete fix in the product, and encourage your comments and input on this ticket: https://trac.nginx.org/nginx/ticket/488 Disclaimer: I work @ NGINX. Thanks, Owen

Double post. See answer here: https://news.ycombinator.com/item?id=11221392
Post reply on HN