Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

trac.nginx.org

Post reply on HN
Print view View original

41–50 of 96 posts

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#41

Post by JensRantil »

kodablah wrote: post #10
JensRantil wrote: post #8

This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this. Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this. Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout r…

"but that will also disable connection timeout retry which is not what you want!" Why is this not what you want? Are you using the reverse proxy as a load balancer to multiple servers? Otherwise, if it's 1:1 proxy (for something like SSL termination) wouldn't having nginx fail/timeout when the server does be acceptable?

Yes, I have multiple servers behind nginx. It's very common.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#42

Post by JensRantil »

muraiki wrote: post #38

Forgive me for not fully understanding this. If I'm just using proxy_pass to a single server (vs using proxy_pass with round robin or using proxy_next_upstream for failover) would this still affect me? In my experience with proxy_pass, a timeout on upstream was reported to the client and the POST was not retried.

No, then you are safe.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#44

Post by mhotchen »

JensRantil wrote: post #8

This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this. Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this. Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout r…

We've hit 3 problems with nginx:

1. Exactly this, we had mystery double trades from our clients and it took us a long time to realise it was nginx assuming we timed out and routing traffic to the next server

2. It doesn't do health checks. When a server goes down it will send 1 out of every 8 real requests to the down server to see if it responds. Having disabled resubmitting of requests to avoid the double trade issue above this means when one of our servers is down, 1 out of every 8 requests will have an nginx proxy error which is significant when you have multiple API calls on a single page

3. This isn't something I've personally hit so can't explain the nitty gritty but it's something one of my coworkers dealt with: outlook webmail does something weird where it opens a connection with a 1GB content size, then sends data continually through that connection, sort of like a push notification hack. Nginx, instead of passing traffic straight through, will collect all data in the response until the response reaches the content size provided in the header (or until the connection is closed). I don't know if nginx is to blame for this one or not, but I do feel that when I send data through the proxy, it should go right through to the client, not be held at the proxy until more data is sent.

HAProxy also solved our issues and is now my go-to proxy. Data goes straight through, it has separate health checks, and it better adheres to HTTP standards. It can also be used for other network protocols which is a bonus.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#45

Post by gshulegaard »

JensRantil wrote: post #8

This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this. Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this. Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout r…

What about `proxy_next_upstream off;`?

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#46

Post by jimjag »

stanleydrew wrote: post #7
SilasX wrote: post #5

Earlier quoted context omitted.

But it's only the POST that this is a problem for, right? PUT and DELETE are supposed to be idempotent so retries are okay, yes?

Yes. Incorrectly implementing the HTTP spec is widespread though, I imagine especially with PUT. People are writing custom HTTP application servers and making PUT do anything and everything. Idempotence doesn't usually enter the picture.

"Yes. Incorrectly implementing the HTTP spec is widespread though"

Huh??? First of all, that's a pretty serious statement to be making with no evidence to support it. Plus, that is no excuse for any server that is NOT compliant.

The very fact that nginx decides to create HTTP status codes, willy-nilly, for its own use shows at least the suspicion that strict compliance is not a priority for them. Thankfully, it is for other web servers.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#47

Post by cbr »

colanderman wrote: post #40
xxs wrote: post #6

Earlier quoted context omitted.

There could be races between PUT/DELETE. There is no grantee how the retry was made.

This is why the If-Match and If-None-Match preconditions exist; they resolve PUT races by checking that the resource is in the expected state.

Well, mostly If-None-Match is used to save bandwidth by allowing the client to validate a stale resource.

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#49

Post by jimjag »

_hyn3 wrote: post #22

This behavior is non-compliant[1] with the RFC. Although (from the standpoint of the RFC), everything on the server side (including nginx itself) is considered the web application, nginx probably takes the implicit position that dealing with multiple requests on non-idempotent methods such as POST is really a problem that the proxied web app itself should cope with. But then nginx puts the web app in an untenable pos…

A pretty good overview of reverse proxies can be found in [1]... spoiler: nginx is NOT one of the best-of-breed. Compliance was/is indeed an issue.

1. http://www.slideshare.net/bryan_call/choosing-a-proxy-server...

Re: Nginx reverse proxies retries PUT/POST/DELETE on response timeout by default

#50

Post by bungle »

mhotchen wrote: post #44
JensRantil wrote: post #8

This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this. Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this. Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout r…

We've hit 3 problems with nginx: 1. Exactly this, we had mystery double trades from our clients and it took us a long time to realise it was nginx assuming we timed out and routing traffic to the next server 2. It doesn't do health checks. When a server goes down it will send 1 out of every 8 real requests to the down server to see if it responds. Having disabled resubmitting of requests to avoid the double trade iss…

Nginx can also be used for other protocols, see stream block.
Post reply on HN