Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ETag and HTTP Caching

rednafi.com

Post reply on HN
Print view View original

21–30 of 92 posts

Re: ETag and HTTP Caching

#21

Post by JaggedJax »

clessg wrote: post #7
JaggedJax wrote: post #2

How is the sample `calculateETag()` function generating a weak ETag? It looks like it will generate a different hash due to any JSON formatting changes. It seems like generating a weak ETag would take more effort since you'd need to either ensure consistent ordering and formatting of the JSON, or generate the Etag on the content before converting it to a JSON string.

That's because it isn't really generating a weak ETag. From the article: > You could make the `calculateETag` function format-agnostic, so the hash stays the same if the JSON format changes but the content does not. The current `calculateETag` implementation is susceptible to format changes, and I kept it that way to keep the code shorter. They seem to agree, a true weak ETag implementation would probably be trickier…

Not sure how I missed that, thanks.

Re: ETag and HTTP Caching

#22

Post by choppaface »

akira2501 wrote: post #6

The ETag can be _anything_. I have an API that serves "files" from a backend storage system. Whenever files are written a revision number is incremented. This is perfect for a weak validator and so my ETags are also blisfully short and semantically useful, typically: ETag: W/"750" This also means the API can just check the revision number and avoid pulling out and decompressing some of the larger payloads that are st…

A great standard but also provides an effective cookie-less mechanism for user tracking e.g.

https://levelup.gitconnected.com/no-cookies-no-problem-using...

Re: ETag and HTTP Caching

#23

Post by infogulch »

ajayvk wrote: post #11

An approach like https://github.com/benbjohnson/hashfs allows file names to be updated at runtime to be content hashed. This removes the need for the extra "304 Not Modified" API calls from the client. This content hash based file renaming is usually done using a build step which renames files. For applications where the static file serving and HTTP request processing are done in the same application, this can be don…

I've also been dissatisfied with http caching not utilizing content hashes enough. If you're using server side templating one issue is that it's not efficient to calculate the hash while you're running the template, it would need to be precalculated to be efficient enough to use.

So I wrote https://github.com/infogulch/xtemplate to scan all assets at startup to precalculate the hash for templates that use it, and if a request comes in with a query parameter ?hash=sha384-xyz and it matches then it gives it a 1 year immutable Cache-Control header automatically. If a file x.ext has a matching x.ext.gz/x.ext.zst/x.ext.br file then (after hashing the content to make sure it matches) client requests that support it are sent a compressed version streamed directly from disk with sendfile2. I call this "Optimal asset serving" (a bit bold perhaps).

Re: ETag and HTTP Caching

#24

Post by EGreg »

codetrotter wrote: post #19
EGreg wrote: post #18

Earlier quoted context omitted.

Wait what? Browsers now offer to load over IPFS? I was wondering how you can trust an IFPS gateway, does does browser verify the file is legit using some checksum? Maybw subresource integrity supports IPFS content hashing or something? How does it generate cid anyway ? https://docs.ipfs.tech/concepts/content-addressing/#cids-are... How would you use SRI here to verify the cid (and not an additional out-of-band hash)…

So in my case, I am not using SRI. But I am using the CID as the name in the path. Using the example file from your link, I would host it as something like https://mywebsite.example.com/ipfs/QmPK1s3pNYLi9ERiq3BDxKa4X... And this was enough for that particular browser I was using to recognize that this file can be attempted to be retrieved directly from IPFS

Yea but the gateway can be compromised, so that is insecure

Re: ETag and HTTP Caching

#25

Post by alganet »

This is nice. It reminds of how miserable my life is.

— Which HTTP code I should return for my API? I already used 404, 403, but I need another one. Damn, HTTP is so old and it makes no sense.

— You can't use HTTP codes like that Bob, they're not a free choice. They're for the protocol, not for your app.

— Let's look at the list. Hm... "412 Precondition Failed". Hey, it sounds nice. It fits to my use case. I'm gonna document it. It means the account is out of balance.

— What is this garbage? Please read the spec. This is going to make our API gateways, CDNs, everything go crazy. Can't let you move on with this PR.

— Look. I documented it, made an enum with the code, it's clean. I'm an experienced REST developer.

— It... it doesn't work like that Bob. Please, read the spec.

— Hey, got enough approvals, "412 Account Out Of Balance" it is! It passes the tests.

For each dev that knows proper HTTP, there's 10.000 Bobs.

Re: ETag and HTTP Caching

#26

Post by esafak »

alganet wrote: post #25

This is nice. It reminds of how miserable my life is. — Which HTTP code I should return for my API? I already used 404, 403, but I need another one. Damn, HTTP is so old and it makes no sense. — You can't use HTTP codes like that Bob, they're not a free choice. They're for the protocol, not for your app. — Let's look at the list. Hm... "412 Precondition Failed". Hey, it sounds nice. It fits to my use case. I'm gonna…

Which article would you ask Bob to read to learn the right way to do it?

Re: ETag and HTTP Caching

#27

Post by AdieuToLogic »

alganet wrote: post #25

This is nice. It reminds of how miserable my life is. — Which HTTP code I should return for my API? I already used 404, 403, but I need another one. Damn, HTTP is so old and it makes no sense. — You can't use HTTP codes like that Bob, they're not a free choice. They're for the protocol, not for your app. — Let's look at the list. Hm... "412 Precondition Failed". Hey, it sounds nice. It fits to my use case. I'm gonna…

For the shortcoming of conveying errors strictly through HTTP status codes, consider:

RFC-7807, Problem Details for HTTP APIs[0]

From the introduction:

  HTTP [RFC7230] status codes are sometimes not sufficient to convey
  enough information about an error to be helpful.  While humans behind
  Web browsers can be informed about the nature of the problem with an
  HTML [W3C.REC-html5-20141028] response body, non-human consumers of
  so-called "HTTP APIs" are usually not.

  This specification defines simple JSON [RFC7159] and XML
  [W3C.REC-xml-20081126] document formats to suit this purpose.  They
  are designed to be reused by HTTP APIs, which can identify distinct
  "problem types" specific to their needs.
HTH

0 - https://datatracker.ietf.org/doc/html/rfc7807

Re: ETag and HTTP Caching

#28

Post by alganet »

esafak wrote: post #26
alganet wrote: post #25

This is nice. It reminds of how miserable my life is. — Which HTTP code I should return for my API? I already used 404, 403, but I need another one. Damn, HTTP is so old and it makes no sense. — You can't use HTTP codes like that Bob, they're not a free choice. They're for the protocol, not for your app. — Let's look at the list. Hm... "412 Precondition Failed". Hey, it sounds nice. It fits to my use case. I'm gonna…

Which article would you ask Bob to read to learn the right way to do it?

Bob has since moved on to crypto, leaving the cache invalidation eternally crippled. New Bob decided that everything is useless and wants to rewrite the whole backend using a faster language.

Re: ETag and HTTP Caching

#29

Post by mozey »

esafak wrote: post #26
alganet wrote: post #25

This is nice. It reminds of how miserable my life is. — Which HTTP code I should return for my API? I already used 404, 403, but I need another one. Damn, HTTP is so old and it makes no sense. — You can't use HTTP codes like that Bob, they're not a free choice. They're for the protocol, not for your app. — Let's look at the list. Hm... "412 Precondition Failed". Hey, it sounds nice. It fits to my use case. I'm gonna…

Which article would you ask Bob to read to learn the right way to do it?

Have to admit I've never used this code, and didn't know what it was about. Quickly read up about it. So ETag is a hash of the resource. You must provide it with requests that modify the resource. If your hash doesn't match the server hash, then 412 Precondition Failed is returned?

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/412

https://www.rfc-editor.org/rfc/rfc9110#status.412

Re: ETag and HTTP Caching

#30

Post by alganet »

AdieuToLogic wrote: post #27
alganet wrote: post #25

This is nice. It reminds of how miserable my life is. — Which HTTP code I should return for my API? I already used 404, 403, but I need another one. Damn, HTTP is so old and it makes no sense. — You can't use HTTP codes like that Bob, they're not a free choice. They're for the protocol, not for your app. — Let's look at the list. Hm... "412 Precondition Failed". Hey, it sounds nice. It fits to my use case. I'm gonna…

For the shortcoming of conveying errors strictly through HTTP status codes, consider: RFC-7807, Problem Details for HTTP APIs[0] From the introduction: HTTP [RFC7230] status codes are sometimes not sufficient to convey enough information about an error to be helpful. While humans behind Web browsers can be informed about the nature of the problem with an HTML [W3C.REC-html5-20141028] response body, non-human consumer…

Good spec.
Post reply on HN