Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updated rate limits for unauthenticated requests

github.blog

Post reply on HN
Print view View original

121–130 of 187 posts

Re: Updated rate limits for unauthenticated requests

#121

Post by notpushkin »

whitehexagon wrote: post #109

If a company the size of MS isn't able handle the DOS caused by the LLM slurpers, then it really is game over for the open internet. We are going to need government approved ID based logins to even read the adverts at this rate. But this feels like a further attempt to create a walled garden around 'our' source code. I say our, but the first push to KYC, asking for phone numbers, was enough for me to delete all and c…

I’m pretty sure they can handle it, but given their continuous (if somewhat bittersweet) relationship with OpenAI, I’m pretty sure they are just trying to protect “their IP“ or something.

Re: Updated rate limits for unauthenticated requests

#122

Post by TheDong »

londons_explore wrote: post #91

Most of these unauthenticated requests are read-only. All of public github is only 21TB. Can't they just host that on a dumb cache and let the bots crawl to their heart's content?

You're talking about the 21TB captured to the arctic code vault, but that 21TB isn't "all of public github"

Quoting from https://archiveprogram.github.com/arctic-vault/

> every *active* public GitHub repository. [active meaning any] repo with any commits between [2019-11-13 and 2020-02-02 ...] The snapshot consists of the HEAD of the default branch of each repository, minus any binaries larger than 100KB in size

So no files larger than 100KB, no commit history, no issues or PR data, no other git metadata.

If we look at this blog post from 2022, the number we get is 18.6 PB for just git data https://github.blog/engineering/architecture-optimization/sc...

Admittedly, that includes private repositories too, and there's no public number for just public repositories, but I'm certain it's at least a noticeable fraction of that ~19PB.

Re: Updated rate limits for unauthenticated requests

#123

Post by dmitrygr »

miohtama wrote: post #104
dmitrygr wrote: post #64

Earlier quoted context omitted.

Criminal charges under CFAA to actual CEOs of actual companies doing this, with long jail terms.

Criminally charging Russian and Chinese does not work. The solution would be to drop these contries off the internet if we want to play hard. The US cannot even stop NSO to hack the system with spyware and Israel is a political ally.

Most of ML training crawlers hitting my site come from USA. Should we drop USA too?

Re: Updated rate limits for unauthenticated requests

#125

Post by graemep »

whitehexagon wrote: post #109

If a company the size of MS isn't able handle the DOS caused by the LLM slurpers, then it really is game over for the open internet. We are going to need government approved ID based logins to even read the adverts at this rate. But this feels like a further attempt to create a walled garden around 'our' source code. I say our, but the first push to KYC, asking for phone numbers, was enough for me to delete all and c…

These exist, and you can self host.

However, a lot of people think Github is the only option, and it benefits from network effects.

Non-profit alternatives suffer from a lack of marketing and deal making. True of most things these days.

Re: Updated rate limits for unauthenticated requests

#126

Post by notpushkin »

graemep wrote: post #125
whitehexagon wrote: post #109

If a company the size of MS isn't able handle the DOS caused by the LLM slurpers, then it really is game over for the open internet. We are going to need government approved ID based logins to even read the adverts at this rate. But this feels like a further attempt to create a walled garden around 'our' source code. I say our, but the first push to KYC, asking for phone numbers, was enough for me to delete all and c…

These exist, and you can self host. However, a lot of people think Github is the only option, and it benefits from network effects. Non-profit alternatives suffer from a lack of marketing and deal making. True of most things these days.

> Non-profit alternatives suffer from a lack of marketing and deal making

Sad but true. I’m trying to promote these whenever I can.

Re: Updated rate limits for unauthenticated requests

#127

Post by hansvm »

mjevans wrote: post #87
zarzavat wrote: post #85

Earlier quoted context omitted.

60/h is obviously well within normal human usage of an app and not bot traffic... A normal rate limit to separate humans and bots would be something like 60 per minute . So it's about an order of magnitude too low.

Use case: crawling possibly related files based on string search hints in a repo you know nothing about... Something on the order of 6 seconds a page doesn't sound TOO out of human viewing range depending on how quickly things load and how fast rejects are identified. I could see ~10 pages / min which is 600 pages / hour. I could also see the argument that a human would get tired at that rate and something closer to…

All of that assuming they're limiting based on human-initiated requests, not the 100x requests actually generated when you click a link.

Re: Updated rate limits for unauthenticated requests

#128

Post by tux3 »

zarzavat wrote: post #81
blinker21 wrote: post #51

Earlier quoted context omitted.

I've hit this over the past week browsing the web UI. For some reason, github sessions are set really short and you don't realise you're not logged in until you get the error message. I really wish github would stop logging me out.

Yes, it's not the rate limits that are the problem per se but GitHub's tendency to log you out and make you go through 2fa. If they would let me stay logged in for a year then I wouldn't care so much.

You might be afflicted with some SSO or enterprise thing, I haven't logged into Github on my personal account in years.

Re: Updated rate limits for unauthenticated requests

#129

Post by kvemkon »

TheDong wrote: post #122
londons_explore wrote: post #91

Most of these unauthenticated requests are read-only. All of public github is only 21TB. Can't they just host that on a dumb cache and let the bots crawl to their heart's content?

You're talking about the 21TB captured to the arctic code vault, but that 21TB isn't "all of public github" Quoting from https://archiveprogram.github.com/arctic-vault/ > every *active* public GitHub repository. [active meaning any] repo with any commits between [2019-11-13 and 2020-02-02 ...] The snapshot consists of the HEAD of the default branch of each repository, minus any binaries larger than 100KB in size So n…

> At GitHub, we store a lot of Git data: more than 18.6 petabytes of it, to be precise.

About $ 250 000 for 1000 HDDs and you get all the data. Meaning private persons such as top FAANG engineers could get a copy of the whole data after 2-3 years job. For companies dealing with AI such raw price is nothing at all.

Re: Updated rate limits for unauthenticated requests

#130

Post by hombre_fatal »

fellerts wrote: post #111
Animats wrote: post #108

Are the scraper sites using a large number of IP addresses, like a distributed denial of service attack? If not, rather than explicit blocking, consider using fair queuing. Do all the requests from IP addresses that have zero requests pending. Then those from IP addresses with one request pending, and so forth. Each IP address contends with itself, so making massive numbers of requests from one address won't cause a…

Huh, that sounds very reasonable, and it's the first time I've heard it mentioned. Why isn't this more wide-spread?

Complex, stateful.

I'm not even sure what that would look like for a huge service like GitHub. Where do you hold those many thousands of concurrent http connections and their pending request queues in a way that you can make decisions on them while making more operational sense than a simple rate limit?

A lot of things would be easy if it were viable to have one big all-knowing giga load balancer.

I remember Rap Genius wrote a blog post whining that Heroku did random routing to their dynos instead of routing to the dyno with the shortest request queue. As opposed to just making an all-knowing infiniscaling giga load balancer that knows everything about the system.

Post reply on HN