Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub is aggressively caching raw.github, breaking many use cases

github.com

Post reply on HN
Print view View original

121–130 of 145 posts

Re: GitHub is aggressively caching raw.github, breaking many use cases

#121

Post by _cenw »

roundandround wrote: post #80
joshspankit wrote: post #74

Earlier quoted context omitted.

Github raw seems like the simplest system for solving cache invalidation: invalidate the cache of a changed file when it’s pushed. They have access to both GitHub and the raw service. I know there are usually all sorts of layers between that make interconnectivity logistically complicated, but am I wrong that at the top-level it’s that simple?

That is too simple for the feature they are using. The client itself has its own cache and the only way to fully prevent traffic from a client is to tell it content it caches will remain valid for some amount of time into the future. For URLs that return the latest entry there is no valid amount of time known in advance by GitHub unless they want to introduce mandatory publication delays. For URLs of specific change…

You could also just commit a file with the intended commit hash, make that the indicator for changes and use the commit in other requests. Has the added benefit that clients only need to fetch a tiny file if nothing changed.

Re: GitHub is aggressively caching raw.github, breaking many use cases

#123

Post by 8note »

Aeolun wrote: post #63
IanCal wrote: post #46

Earlier quoted context omitted.

The commit ID seems to be part of the URL you can use https://raw.githubusercontent.com/burekasKodi/repository.bur... I'm fine with them caching "latest" tbh.

I’m not fine with latest not actually being latest. That defeats the point of the URL.

There's a hard limit on that - the speed of light.

You can't know what the actual latest is, only some cached value. The actual value may have changed while the message is in flight

Re: GitHub is aggressively caching raw.github, breaking many use cases

#124

Post by low_tech_punk »

nubinetwork wrote: post #14

Wouldn't an aggressive cache mean it has the most recent content, not the other way around?

You mean aggressive cache eviction/invalidation? I think they might be taking about aggressive caching. Subtle difference in words.

Re: GitHub is aggressively caching raw.github, breaking many use cases

#125

Post by 8note »

iLoveOncall wrote: post #68
EdwardDiego wrote: post #54

Earlier quoted context omitted.

Ridiculous is expecting a service you're not paying for to serve files in a way that fits your use case when you've never entered into a contract that guarantees the behaviour you're relying on. You can use Git just fine without GitHub.

So expecting that a raw file isn't outdated is a special usecase now? And I don't see where you see that I'm not a paying user? This issue affects every repository, including ones where the user pays.

The CAP theorem certainly isn't new, and it's unreasonable to think GitHub has solved it

Re: GitHub is aggressively caching raw.github, breaking many use cases

#126

Post by remram »

ilyt wrote: post #117
roundandround wrote: post #80

Earlier quoted context omitted.

That is too simple for the feature they are using. The client itself has its own cache and the only way to fully prevent traffic from a client is to tell it content it caches will remain valid for some amount of time into the future. For URLs that return the latest entry there is no valid amount of time known in advance by GitHub unless they want to introduce mandatory publication delays. For URLs of specific change…

Tag it with etag = hash, done. The client side isn't hard part. The server side would require pushing any invalidation to (I imagine) whole tree of caches, which isn't exactly that hard if you plan for it from the start and have some way of upstream telling downstream file changes, but, well, they probably don't as I'd imagine they didn't expected people to pin their infrastructure to some binary blob on github that…

Etag doesn't let you "fully prevent traffic from a client" (GP's exact words). They'll still send a request to which you need to reply with a 304 after checking the resource.

Re: GitHub is aggressively caching raw.github, breaking many use cases

#127

Post by jakear »

taeric wrote: post #114
jakear wrote: post #103

Earlier quoted context omitted.

And how do you check what the new name of the content you haven't seen is?

Serious question? There will be a root object that has a stable name through updates. When you write that, you embed current content hashes in it. Someone could get the cached of it, of course. But since it content addresses what it links to, it should avoid incoherent groups of cached data.

I agree this would provide for increased coherency, but the additional indirection is hardly ideal. And really it's just kicking the caching problem up to a super-object that will probably be thrashed much harder than any individual file, and may be larger than each individual file.

All in all it's a "certified hard problem", there's a bunch of domain specific nuance an HN thread couldn't hope to capture.

Re: GitHub is aggressively caching raw.github, breaking many use cases

#128

Post by taeric »

jakear wrote: post #127
taeric wrote: post #114

Earlier quoted context omitted.

Serious question? There will be a root object that has a stable name through updates. When you write that, you embed current content hashes in it. Someone could get the cached of it, of course. But since it content addresses what it links to, it should avoid incoherent groups of cached data.

I agree this would provide for increased coherency, but the additional indirection is hardly ideal. And really it's just kicking the caching problem up to a super-object that will probably be thrashed much harder than any individual file, and may be larger than each individual file. All in all it's a "certified hard problem", there's a bunch of domain specific nuance an HN thread couldn't hope to capture.

The extra indirection may or may not be a problem, of course. For a lot of places this is used, it is "index.html," after all.

That said, I do find folks bend way too many things into fitting this pattern; such that I do not mean to be dismissive of the criticism.

Post reply on HN