Oh this topic can cost you some years of your life
ETag and HTTP Caching
41–50 of 92 posts
Re: ETag and HTTP Caching
#42Earlier quoted context omitted.
Ah, the eternal battle. Do you return an HTTP error code if the problem lies in the domain? I think I've seen everything on that scale. On one end: { status: 200, error: InsufficientFunds } On the other: - "let's use 409, it perfectly fits our use case" + "but we don't have useful error codes for all the other domain errors."
Status 200 with "InsufficientFunds" can be correct. Let's assume your resource is "/account/1234/withdraw-availability" It's a hypothetical endpoint you can GET to know if you can withdraw money. You hit it, and the request is sucessful (the server understood and will inform you whether withdrawaw is available or not). Let's assume your resource is "/account/1234/withdraw" This other hypothetical endpoint you can POS…
One more example, if your request contains a batch of operations, you generally have to return 200, or maybe 204, if it was successfully received and should not be retried in full. In the response body, you might give other response codes for specific errors or failures to retry in a new request. So it can easily make sense to return 200 when there are e.g. partial failures and partial success and the request was properly formatted, authorized and acted upon.
Re: ETag and HTTP Caching
#43I've not seen a very convincing use-case for ETags vs Last-Modified date caching.
In the example request, the server still has to do all of the work generating the page, in order to calculate the ETag and then determine whether or not the page has changed. In most situations, it's simpler to have timestamps to compare against, because that gives the server a faster way to spot unmodified data.
e.g. you get a HTTP request for some data that you know is sourced from a particular file, or a DB table. If the client sends a If-Modified-Since (or whatever the header name is), you have a good chance to be able to check the modified time of the data source before doing any complicated data processing, and are able to send back a not modified response sooner.
Re: ETag and HTTP Caching
#44How is the sample `calculateETag()` function generating a weak ETag? It looks like it will generate a different hash due to any JSON formatting changes. It seems like generating a weak ETag would take more effort since you'd need to either ensure consistent ordering and formatting of the JSON, or generate the Etag on the content before converting it to a JSON string.
That's because it isn't really generating a weak ETag. From the article: > You could make the `calculateETag` function format-agnostic, so the hash stays the same if the JSON format changes but the content does not. The current `calculateETag` implementation is susceptible to format changes, and I kept it that way to keep the code shorter. They seem to agree, a true weak ETag implementation would probably be trickier…
I could probably write one pretty quick, under the assumption that we are storing only JS-compatible JSON with no encoding hiccups (JSON sadly isn’t as standard as it appears at first glance…) just hash(JSON.stringify(JSON.parse(fileText))) and you’re done. This assumes the same parse and serialize methods are expected to be used at both ends, that they only normalize formatting and that you don’t have to worry about number representation doing weird things. I wouldn’t actually sort keys as sorting is technically a change in behavior and good browsers today do not re-order object keys for you, though your code can do that, of course. I considered skipping the second JSON serialize, but it makes a buffer out of an object so it’s easy enough to use. One could imagine a more efficient approach would modify the hashing to occur against buffer chunks of the JSON, but intentionally skip the whitespace. This avoids unintentional data serialization but obviously the parsing routine would have to match the recipient exactly to work correctly. And it still assumes you’re receiving oddly formatted but valid JSON, which doesn’t sound like a safe assumption to make. If your JSON varies in format I wouldn’t ever want to assume I’d be able to parse it correctly. I mean, what if a return character slips in by mistake amongst all the newlines?
Re: ETag and HTTP Caching
#45if only browsers respected this. none of browsers use ETag and If-None-Match mechanism. instead they do their own wizzardy caching...
Re: ETag and HTTP Caching
#46if only browsers respected this. none of browsers use ETag and If-None-Match mechanism. instead they do their own wizzardy caching...
I was refactoring a project serving user uploaded files yesterday, and had the occasion to test caching. Both Firefox and Chrome used ETag and If-None-Match properly to check and cache queries. Which problems did you encounter?
There was still one thing that surprised me a bit (but also makes sense). Images are fetched only once per page load in my testing. If an image with 60sec of cache is loaded, then removed by JS and added back after 2 minutes, then the browser will reuse the image from the initial load.
Re: ETag and HTTP Caching
#47Earlier quoted context omitted.
That's because it isn't really generating a weak ETag. From the article: > You could make the `calculateETag` function format-agnostic, so the hash stays the same if the JSON format changes but the content does not. The current `calculateETag` implementation is susceptible to format changes, and I kept it that way to keep the code shorter. They seem to agree, a true weak ETag implementation would probably be trickier…
I could probably write one pretty quick, under the assumption that we are storing only JS-compatible JSON with no encoding hiccups (JSON sadly isn’t as standard as it appears at first glance…) just hash(JSON.stringify(JSON.parse(fileText))) and you’re done. This assumes the same parse and serialize methods are expected to be used at both ends, that they only normalize formatting and that you don’t have to worry about…
Since the author wrote in go, an approximate equivalent would sort the keys by default (assuming you decode into a map and Marshall that).
Re: ETag and HTTP Caching
#48This is nice. It reminds of how miserable my life is. — Which HTTP code I should return for my API? I already used 404, 403, but I need another one. Damn, HTTP is so old and it makes no sense. — You can't use HTTP codes like that Bob, they're not a free choice. They're for the protocol, not for your app. — Let's look at the list. Hm... "412 Precondition Failed". Hey, it sounds nice. It fits to my use case. I'm gonna…
To be fair, to me HTTP looks like: - The first line of an HTTP request has its own format (space-separated-ish), and mixes method and URL path. - The URL path in that first line has its own format and weird escaping, and mixes one path with zero-or-more key value pairs. - The headers have their own format. - The body has its own format. Most (all?) HTTP libraries for clients and servers abstract all that mess away in…
URLs are opaque:
- https://www.w3.org/2000/12/drm-ws/pp/connolly/slide8-0.html
- https://www.w3.org/DesignIssues/Axioms.html#opaque
HTTP header format goes back to ARPA times. Email reused them, so did HTTP.
- https://datatracker.ietf.org/doc/html/rfc822#section-3.2
Many of these choices are there for backwards compatibility and reuse. I can totally understand why.
An HTTP body has no predefined format. It can be anything. It can be a stream (HTTP into WebSocket upgrade, for example). It is the media type that defines how the body should be interpreted.
HTTP requests are meant to be used before they are fully transmitted, and are formtted in a way to leverage socket communication. JSON, on the other hand, needs the whole document to be read before it can be safely interpreted.
HTTP has more moving parts, but it also does so much more. These two aren't even comparable, they're not in the same layer.
I understand the urge to "improve" on all of this "legacy", however, one must consider how much was built upon these standards and if there's anything real to gain by changing them.
Re: ETag and HTTP Caching
#49This is nice. It reminds of how miserable my life is. — Which HTTP code I should return for my API? I already used 404, 403, but I need another one. Damn, HTTP is so old and it makes no sense. — You can't use HTTP codes like that Bob, they're not a free choice. They're for the protocol, not for your app. — Let's look at the list. Hm... "412 Precondition Failed". Hey, it sounds nice. It fits to my use case. I'm gonna…
I'm using 400 for everything, to the hell with it.
Re: ETag and HTTP Caching
#50I recently implemented this, great write-up. Regarding the hashing function, I’m curious about opinions. In my implementation I went for a cheap but weak cryptographic hash at first. Then I got worried that some auditor would flag it and time would be wasted convincing them to change their mind. But then I stumbled upon FNV [1], a non-cryptographic hash and part of Go’s standard library and went for it. Any thoughts?…
I recommend checking out XXHash[0], FNV is simple but not really optimized and relatively low quality (often still good enough). From the readme page:
Hash Name, Bandwidth, Small Velocity, Quality, Comment
XXH3 (SSE2), 31.5 GB/s, 133.1, 10,
FNV64, 1.2 GB/s, 62.7, 5, Poor avalanche properties
[0]: https://github.com/Cyan4973/xxHash
Also ETag is exactly the kind of thing non-cryptographic hashes are meant for, but if you can't convince them Blake3 is a very fast modern cryptographic hash function.