Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default • Hacker News Board

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#21

Post by dragonwriter » Thu, Mar 03, 2016, 4:48 PM UTC

SilasX wrote: post #11

xxs wrote: post #6

Earlier quoted context omitted.

There could be races between PUT/DELETE. There is no grantee how the retry was made.

A race shouldn't matter so long as both succeed at least once; that would have the same effect as if either/both had succeeded multiple times. The only difference is whether the user is informed of whether a related action obviated their request, which is going to happen anyway. Edit: turns out I was wrong and assumed PUT should fail if the resource doesn't exist, which isn't how it works. (Probably because of writin…

> A race shouldn't matter so long as both succeed at least once; that would have the same effect as if either/both had succeeded multiple times.

Idempotence only means that the same single method repeated additional times on its own will not produce different end states. It doesn't necessarily guarantee this for combinations of methods in different orderings.

It doesn't stop PUT/DELETE/PUT/DELETE from having different results than PUT/DELETE/DELETE/PUT to the same resource. (You can do assure that these are equivalent in a particular HTTP-compliant application, but it goes beyond the base semantics of HTTP to do so.)

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#22

Post by _hyn3 » Thu, Mar 03, 2016, 4:52 PM UTC

This behavior is non-compliant[1] with the RFC.

Although (from the standpoint of the RFC), everything on the server side (including nginx itself) is considered the web application, nginx probably takes the implicit position that dealing with multiple requests on non-idempotent methods such as POST is really a problem that the proxied web app itself should cope with.

But then nginx puts the web app in an untenable position. Consider the example of non-idempotent POST to create a new user account. The new user account includes a username, email address, and password. Because it's proxied, nginx creates a duplicate request for this new user account in the circumstances described in this bug report.

How should the web app deal with the duplicate request?

a. Accept the first request (200 OK) and decline the second request since the account was already created (i.e., 409 Conflict), or

b. Create two duplicate user accounts (200 OK for both)

Obviously, the ONLY correct response is the first one, but what happens next is really up to nginx: will the client receive the 409 Conflict (etc) or will it receive the 200?

Well, who knows?! It's completely indeterminate.

If the client gets the 200 OK, great. But what if it doesn't? These duplicate requests seem like they could lead to an nginx race condition as well. And what gets logged?

This behavior clearly violates both the spirit and the letter of RFC 7231 (as well as being an obviously poor engineering decision!).

Note also the long time (years!)[2] that this has been a known, outstanding bug without any action taken. Another commenter actually said this caused cascading failure in their application that killed their app.

Bottom line... nginx is a great, fast static server, but definitely not a good proxy for dynamic apps. We're trying to figure out how fast we can migrate Userify (plug: SSH key management for EC2)[3] from nginx to HA-Proxy, since we use it to front-end our REST API.

1. https://tools.ietf.org/html/rfc7231#page-23

2. https://trac.nginx.org/nginx/ticket/488#comment:3

3. https://userify.com

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#23

Post by stormbrew » Thu, Mar 03, 2016, 4:54 PM UTC

_hyn3 wrote: post #22

This behavior is non-compliant[1] with the RFC. Although (from the standpoint of the RFC), everything on the server side (including nginx itself) is considered the web application, nginx probably takes the implicit position that dealing with multiple requests on non-idempotent methods such as POST is really a problem that the proxied web app itself should cope with. But then nginx puts the web app in an untenable pos…

By default[1], nginx only talks to backends in http/1.0, so the operative rfc is (sadly) https://tools.ietf.org/html/rfc1945. Though it did establish GET/HEAD as safe and other methods as not, the idea of idempotence itself was not yet present and it doesn't have any language I'm aware of to restrict client retries on non-safe methods.

That said, I don't know if nginx does any better if you set it to http/1.1 mode on this issue. I assume not, to be honest.

[1] http://nginx.org/en/docs/http/ngx_http_proxy_module.html#pro...

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#24

Post by SilasX » Thu, Mar 03, 2016, 4:55 PM UTC

dragonwriter wrote: post #21

SilasX wrote: post #11

Earlier quoted context omitted.

A race shouldn't matter so long as both succeed at least once; that would have the same effect as if either/both had succeeded multiple times. The only difference is whether the user is informed of whether a related action obviated their request, which is going to happen anyway. Edit: turns out I was wrong and assumed PUT should fail if the resource doesn't exist, which isn't how it works. (Probably because of writin…

> A race shouldn't matter so long as both succeed at least once; that would have the same effect as if either/both had succeeded multiple times. Idempotence only means that the same single method repeated additional times on its own will not produce different end states. It doesn't necessarily guarantee this for combinations of methods in different orderings. It doesn't stop PUT/DELETE/PUT/DELETE from having differen…

I was only saying there that the combination didn't create other problems (due to race conditions), not that that fact was related to idempotence. Though it happens to be true for the combination of PUT/DELETE as well!

I think you're equating my claims about what methods are idempotent with my claims about what reorderings matter.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#25

Post by SilasX » Thu, Mar 03, 2016, 4:57 PM UTC

Someone wrote: post #18

SilasX wrote: post #11

Earlier quoted context omitted.

A race shouldn't matter so long as both succeed at least once; that would have the same effect as if either/both had succeeded multiple times. The only difference is whether the user is informed of whether a related action obviated their request, which is going to happen anyway. Edit: turns out I was wrong and assumed PUT should fail if the resource doesn't exist, which isn't how it works. (Probably because of writin…

Shouldn't matter? DELETE foo/bar PUT foo/bar If that delete gets a retry, actual execution order could be PUT foo/bar DELETE foo/bar Or am I misunderstanding this?

It doesn't matter: as I said, that has the same end state (no foo/bar resource), the only possible difference is response code i.e. whether (in this case) you get to learn that your update doesn't matter.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#26

Post by [deleted] » Thu, Mar 03, 2016, 4:59 PM UTC

stormbrew wrote: post #23

_hyn3 wrote: post #22

This behavior is non-compliant[1] with the RFC. Although (from the standpoint of the RFC), everything on the server side (including nginx itself) is considered the web application, nginx probably takes the implicit position that dealing with multiple requests on non-idempotent methods such as POST is really a problem that the proxied web app itself should cope with. But then nginx puts the web app in an untenable pos…

By default[1], nginx only talks to backends in http/1.0, so the operative rfc is (sadly) https://tools.ietf.org/html/rfc1945 . Though it did establish GET/HEAD as safe and other methods as not, the idea of idempotence itself was not yet present and it doesn't have any language I'm aware of to restrict client retries on non-safe methods. That said, I don't know if nginx does any better if you set it to http/1.1 mode o…

[deleted]

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#27

Post by SilasX » Thu, Mar 03, 2016, 5:01 PM UTC

lukeschlather wrote: post #14

SilasX wrote: post #5

Earlier quoted context omitted.

But it's only the POST that this is a problem for, right? PUT and DELETE are supposed to be idempotent so retries are okay, yes?

Sure, but the point is you should design your POSTs to be idempotent as well.

A better approach IMHO is to turn them into PUTs. If something would normally be a POST but you've eg used GUIDs to ensure that creation actions are idempotent, then such actions should be PUTs.

But then again, some things must not be idempotent: eg "shuffle this deck of cards in an order that is random to me".

Edit: On second thought, you could make that idempotent too, albeit at the cost of increasing server load and your app's architecture's complexity -- you would just have to verify that the deck has had some reordering since that client's request, and not make any further reorderings in response to that client, since from their perspective it's still randomized.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#28

Post by dragonwriter » Thu, Mar 03, 2016, 5:02 PM UTC

stormbrew wrote: post #23

_hyn3 wrote: post #22

This behavior is non-compliant[1] with the RFC. Although (from the standpoint of the RFC), everything on the server side (including nginx itself) is considered the web application, nginx probably takes the implicit position that dealing with multiple requests on non-idempotent methods such as POST is really a problem that the proxied web app itself should cope with. But then nginx puts the web app in an untenable pos…

By default[1], nginx only talks to backends in http/1.0, so the operative rfc is (sadly) https://tools.ietf.org/html/rfc1945 . Though it did establish GET/HEAD as safe and other methods as not, the idea of idempotence itself was not yet present and it doesn't have any language I'm aware of to restrict client retries on non-safe methods. That said, I don't know if nginx does any better if you set it to http/1.1 mode o…

> By default[1], nginx only talks to backends in http/1.0, so the operative rfc is (sadly) https://tools.ietf.org/html/rfc1945. Though it did establish POST/PUT/etc. as 'safe'

No, only GET and HEAD are safe in RFC 1945.

> the idea of idempotence itself was not yet present and it doesn't have any language I'm aware of to restrict client retries on non-safe methods.

That actually doesn't really change the situation that much: without an idempotence guarantee, there is no protocol-level basis for a proxy (reverse or otherwise) to assume that a non-safe method is repeatable. Under HTTP 1.0, by the RFC alone, there's no justification for treating anything other than GET or HEAD as reliably repeatable. (Except perhaps that the operations described by PUT and DELETE are at least arguably, as specified, idempotent, even though the term is not invoked and the guarantee is not made express.)

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#29

Post by joosters » Thu, Mar 03, 2016, 5:04 PM UTC

lukeschlather wrote: post #14

SilasX wrote: post #5

Earlier quoted context omitted.

But it's only the POST that this is a problem for, right? PUT and DELETE are supposed to be idempotent so retries are okay, yes?

Sure, but the point is you should design your POSTs to be idempotent as well.

Well, they are defined as non-idempotent, so there's no reason why you 'should' design this way. You can't make every request idempotent.

It's best to design so that duplicate POSTs are handled sensibly (e.g. you don't make a user pay for the same product twice), but the response to the second POST is unlikely to be the same as the first one, so they aren't idempotent.

More difficult cases are where an action could legitimately be performed twice, e.g. adding an item to a shopping cart. You must differentiate between a wrongly-duplicated request, and a real request to add the second item. One way to do this is to add parameters to the POST so that it can be identified as a duplicate. But it can be tricky to do this without holding a lot of extra state in the server application, and there are all sorts of concurrency problems when you have a cluster of servers.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#30

Post by marcosdumay » Thu, Mar 03, 2016, 5:10 PM UTC

kodablah wrote: post #10

JensRantil wrote: post #8

This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this. Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this. Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout r…

"but that will also disable connection timeout retry which is not what you want!" Why is this not what you want? Are you using the reverse proxy as a load balancer to multiple servers? Otherwise, if it's 1:1 proxy (for something like SSL termination) wouldn't having nginx fail/timeout when the server does be acceptable?

> Are you using the reverse proxy as a load balancer to multiple servers?

That's extremely likely.

Somebody with an Nginx reverse proxy is probably using it for high availability, load balancing and static files cache, probably at the same time. This is what it is good for.