81–90 of 96 posts
This is why idempotence is so important.
This essentially became the laser of death the other day and lead to cascading failure which eventually brought down our system. That's why I'm posting this. Very few people know about this and it's really scary. I'm happy people are voting this up to increase some awareness of this. Potential workarounds: You might think disabling `proxy_next_upstream timeout` will do, but that will also disable connection timeout r…
We've hit 3 problems with nginx: 1. Exactly this, we had mystery double trades from our clients and it took us a long time to realise it was nginx assuming we timed out and routing traffic to the next server 2. It doesn't do health checks. When a server goes down it will send 1 out of every 8 real requests to the down server to see if it responds. Having disabled resubmitting of requests to avoid the double trade iss…
Earlier quoted context omitted.
Cool. Did you ever consider patching nginx upstream instead?
Yes. Before implementing this module we contacted nginx developers and they didn't think it is a problem. This is why we had to create our own module.
This is why idempotence is so important.
Well, its why adherence to the semantics of the HTTP spec (which this behavior is an example of failing ) is important: POST is not defined to be idempotent, so nothing should act by default as if POSTs are repeatable.
But there's a layer beneath HTTP as well. If all you get back is a TCP RST, did the request succeed or fail? How about if you get an ICMP unreachable or just a timeout ... should you retry?
So, the Internet being what it is, it is probably not a bad idea to aim for idempotence for the critical bits.
How is one supposed to take seriously web infrastructure software that exhibits such a basic failure of understanding core web standards? From even a cursory reading of the HTTP RFCs one will understand that "POST = unsafe = don't retry after request sent = return 504 on reply timeout". I mean, a bug's a bug; but this was known for two years!
> How is one supposed to take seriously web infrastructure software that exhibits such a basic failure of understanding core web standards? How? Probably based on the fact that otherwise it's a frigging great app that powers like 15% of the web, including some of the biggest sites out there.
Earlier quoted context omitted.
If-None-Match: * ensures that another client hasn't created the resource you are trying to create. It is equally important as If-Match for resolving race conditions.
It's fairly uncommon to use PUT for resource creation. In that case, however, if the server supported it, yes you could use If-None-Match. I really have to wonder about the architecture of the system, however, if two clients can simultaneously decide to create the same resource rather than two similar resources.
* Literally the first thing RFC 7231 says about PUT is "The PUT method requests that the state of the target resource be created or replaced […]". RFC 7231 takes into account many changes in HTTP practice over the past decade (even bizarre ones like POST-to-GET on a 301 redirect); if create-on-PUT were frowned upon, it would be called out.
* PUT as described in RFC 7231 is the same thing as UPSERT in an RDBMS, or a write operation in a key-value store. These are certainly not uncommon DB operations; their REST analogue is similarly useful.
Here's some examples:
* PUT is how documents are created in WebDAV. WebDAV is multi-user, so two users may decide to create a document with the same name, just like on any file system. If-None-Match: * is the only way to support the O_EXCL flag on POSIX open(2).
* A resource which represents attributes of arbitrary external resources will have a URI named after the external resource (e.g. UPC or SHA-1, etc.), and therefore must be created with PUT. If-None-Match: * is the only way to prevent lost updates when the external resource is first made known to the system.
PUT-as-create is sound design supported by precedent for any system where the keys have a priori meaning.
Earlier quoted context omitted.
We've hit 3 problems with nginx: 1. Exactly this, we had mystery double trades from our clients and it took us a long time to realise it was nginx assuming we timed out and routing traffic to the next server 2. It doesn't do health checks. When a server goes down it will send 1 out of every 8 real requests to the down server to see if it responds. Having disabled resubmitting of requests to avoid the double trade iss…
3. is the reason why NGinX is the recommended proxy in front of webapps with scarce parallelism (for example Ruby with Unicorn; see http://unicorn.bogomips.org/PHILOSOPHY.html for an explanation) when "slow clients" are to be expected. NGinX is protecting the webapp from blocked workers by slow clients and Outlook Webmail seems to behave just like one. I don't know by heart how to tune this behavior if one wants to a…
This applies to uWSGI (uwsgi_pass) as well, right?