Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updated rate limits for unauthenticated requests

github.blog

Post reply on HN
Print view View original

101–110 of 187 posts

Re: Updated rate limits for unauthenticated requests

#101

Post by tonyhart7 »

grishka wrote: post #90
mjevans wrote: post #88

Earlier quoted context omitted.

Static, Near Static (not generated on demand at least; generated only on real content update), and Login seems likely. AI not caching things is a real issue. Sites being difficult TO cache / failing the 'wget mirror test' is the other side of the issue.

What about AI not respecting robots.txt? I myself have never ran into this, but I've seen complaints of many people who did.

"What about AI not respecting robots.txt?"

since when actor that want gather your entire data respect things like this??? how can you enforce such things with just "please don't crawl this directory thanks"

Re: Updated rate limits for unauthenticated requests

#102

Post by tonyhart7 »

watermelon0 wrote: post #8

Time for Mozilla (and other open-source projects) to move repositories to sourcehut/Codeberg or self-hosted Gitlab/Forgejo?

time to move (alternative github experience) is works, until those crawler goes into alternative then forcing those rate limit as well

Re: Updated rate limits for unauthenticated requests

#103

Post by miohtama »

jhgg wrote: post #53

The truth is this won't actually stop AI crawlers and they'll just move to a large residential proxy pool to work around it. Not sure what the solution is honestly.

At GitHub scale, crawlers will run out of IP addresses regardless if they are use residential addresses.

Re: Updated rate limits for unauthenticated requests

#104

Post by miohtama »

dmitrygr wrote: post #64
jhgg wrote: post #53

The truth is this won't actually stop AI crawlers and they'll just move to a large residential proxy pool to work around it. Not sure what the solution is honestly.

Criminal charges under CFAA to actual CEOs of actual companies doing this, with long jail terms.

Criminally charging Russian and Chinese does not work. The solution would be to drop these contries off the internet if we want to play hard.

The US cannot even stop NSO to hack the system with spyware and Israel is a political ally.

Re: Updated rate limits for unauthenticated requests

#105

Post by arkh »

soraminazuki wrote: post #97
jonas21 wrote: post #95

Earlier quoted context omitted.

Just sign in, problem solved. It baffles me that a site can provide a useful service that costs money to run, and all you need to do to use it is create a free account -- and people still find that egregious.

That's not how consent works. GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible.

Who could have known that Microsoft would pull some shenanigans?

Is 20 years too long ago to learn from then?

Embrace. Extend. Extinguish. This has never gone away.

Re: Updated rate limits for unauthenticated requests

#106

Post by miohtama »

voidnap wrote: post #7
pogue wrote: post #4

I assume they're trying to keep ai bots from strip mining the whole place. Or maybe your IP/browser is questionable.

I encountered this on github last week. Very agressive rate limiting. My browser and IP is very ordinary. Since Microsoft is struggling to make ends meet, maybe they could throw a captcha or proof of work like Anubis by xe iaso. They already disabled code search for unauthenticated users. Its totally plausible they will disable code browsing as well.

Just sign in if it's an issue for your usage.

Re: Updated rate limits for unauthenticated requests

#107

Post by arkh »

tonyhart7 wrote: post #101
grishka wrote: post #90

Earlier quoted context omitted.

What about AI not respecting robots.txt? I myself have never ran into this, but I've seen complaints of many people who did.

"What about AI not respecting robots.txt?" since when actor that want gather your entire data respect things like this??? how can you enforce such things with just "please don't crawl this directory thanks"

> how can you enforce such things

A van, some balaclavas and 4 people with big sticks.

Re: Updated rate limits for unauthenticated requests

#108

Post by Animats »

Are the scraper sites using a large number of IP addresses, like a distributed denial of service attack? If not, rather than explicit blocking, consider using fair queuing. Do all the requests from IP addresses that have zero requests pending. Then those from IP addresses with one request pending, and so forth. Each IP address contends with itself, so making massive numbers of requests from one address won't cause a problem.

I put this on a web site once, and didn't notice for a month that someone was making queries at a frantic rate. It had zero impact on other traffic.

Re: Updated rate limits for unauthenticated requests

#109

Post by whitehexagon »

If a company the size of MS isn't able handle the DOS caused by the LLM slurpers, then it really is game over for the open internet. We are going to need government approved ID based logins to even read the adverts at this rate.

But this feels like a further attempt to create a walled garden around 'our' source code. I say our, but the first push to KYC, asking for phone numbers, was enough for me to delete all and close my account. Being on the outside, it feels like those walls get taller every month. I often see an interesting project mentioned on HN and clone the repo, but more and more times that is failing. Trying to browse online is now limited, and they recently disabled search without an account.

For such a critical piece of worldwide technology infrastructure, maybe it would be better run by a not-for-profit independent foundation. I guess, since it is just git, anyone could start this, and migration would be easy.

Re: Updated rate limits for unauthenticated requests

#110

Post by Dylan16807 »

globie wrote: post #100
roughly wrote: post #69

Earlier quoted context omitted.

AI isn't the one closing up shop, it’s the one looting all the stores and taking everything that isn’t bolted down. The AI companies are bad actors that are exploiting the openness of the internet in a fashion that was obviously going to lead to this result - the purpose of these scrapers is to grab everything they can and repackage it into a commercial product which doesn’t return anything to the original source. Of…

This rings the same tune as the MPAA and RIAA utilizing lawfare to destroy freedom online when pirates were the ones "break[ing] the internet." Could you help me understand what the difference is between your point and the arguments MPAA and RIAA used to ruin the torrent users' lives they concluded were "thieves"? As a rule of thumb, do you think people who are happy with the services they contribute content to being…

> Could you help me understand what the difference is

Well the main difference is that this is being used to justify blocking and not demanding thousands of dollars.

> When AI can perfectly replicate the browsing behavior of a human being

They're still being jackasses because I'm willing to pay to give free service to X humans but not 20X bots pretending to be humans.

Post reply on HN