161–170 of 187 posts
If a company the size of MS isn't able handle the DOS caused by the LLM slurpers, then it really is game over for the open internet. We are going to need government approved ID based logins to even read the adverts at this rate. But this feels like a further attempt to create a walled garden around 'our' source code. I say our, but the first push to KYC, asking for phone numbers, was enough for me to delete all and c…
you mean https://savannah.gnu.org ?
Why can't people harden their software with guards? Proper DDoS protection? Better caching? Rewrite the hot paths in C, Rust, Zig, Go, Haskell etc.?
It strikes me as very odd, the atmosphere of these threads. So much doom and gloom. If my site was hit by an LLM scraper I'd be like "oh, it's on!", a big smile, and I'll get to work right away. And I'll have that work approved because I'll use the occasion to convince the executives of the need. And I'll have tons of fun.
Can somebody offer a take on why are we, the forefront of the tech sector, just surrendering almost without a single shot?
Earlier quoted context omitted.
Embrace, extend, extinguish.
… I… what has been embraced, extended and extinguished? I see no MS or GitHub specific extension, here. Copilot exists, and so do many other tools. Copilot can use lots of non-Microsoft models, too, including models from non-Microsoft companies. You can also get git repository hosting from other companies. You can even do it yourself. So, explain yourself. What has been embraced, extended, and extinguished? Be specif…
A take that I'm not seeing in all the "LLM scrapers are heading to our site, run for your lives!" threads is this: Why can't people harden their software with guards? Proper DDoS protection? Better caching? Rewrite the hot paths in C, Rust, Zig, Go, Haskell etc.? It strikes me as very odd, the atmosphere of these threads. So much doom and gloom. If my site was hit by an LLM scraper I'd be like "oh, it's on!", a big s…
I assume they're trying to keep ai bots from strip mining the whole place. Or maybe your IP/browser is questionable.
What's being strip mined is the openness of the Internet, and AI isn't the one closing up shop. Github was created to collaborate on and share source code. The company in the best position to maximize access to free and open software is now just a dragon guarding other people's coins. The future is a .txt file of John Carmack pointing out how efficient software used to be, locked behind a repeating WAF captcha, forev…
A take that I'm not seeing in all the "LLM scrapers are heading to our site, run for your lives!" threads is this: Why can't people harden their software with guards? Proper DDoS protection? Better caching? Rewrite the hot paths in C, Rust, Zig, Go, Haskell etc.? It strikes me as very odd, the atmosphere of these threads. So much doom and gloom. If my site was hit by an LLM scraper I'd be like "oh, it's on!", a big s…
Because our sites are written in layers of abstraction and terrible design, which leads to requests taking serious server resources. If we hosted everything "well", you'd get a few 10-20k req/s per CPU core, but we aren't.
Especially when the car's plastic suspension is costing them extra money? I don't get it here, for real. I would think that selfish capitalistic interests would have them come around at one point! (Clarification: invest $5M for a year before the whole thing starts costing you extra $30M a year, for example.)
And don't even get me started on the fact that GitHub is written in one of the most hardware-inefficient web frameworks (Rails). I mean OK, Rails is absolutely great for many things because most people's business will never scale as much and as such the initial increased developer velocity is an unquestionable one-sided win. I get that and I stopped hating Rails long time ago (even though I dislike it; but I do recognize where it's a solid and even preferred choice). But I've made a lot of money from trying to modernize and maintain Rails monoliths; it's just not suited for one scale and on -- without paying for extremely expensive consultants that is. It's like, everything can be made to work but it does start costing exponentially more from one scale and further up.
And yet nobody at GitHub figures "Maybe it's time we rewrite some of the hot paths?" or just "Make more aggressive caching even if it means some users see data outdated by 30 seconds or so"? Nothing at all?
Sorry, I am kind of ranting and not really saying anything to you per se. I am just very puzzled about how paralyzed GitHub seems under Microsoft.
Earlier quoted context omitted.
This rings the same tune as the MPAA and RIAA utilizing lawfare to destroy freedom online when pirates were the ones "break[ing] the internet." Could you help me understand what the difference is between your point and the arguments MPAA and RIAA used to ruin the torrent users' lives they concluded were "thieves"? As a rule of thumb, do you think people who are happy with the services they contribute content to being…
The moral argument in favor of piracy was that it didn’t cost the companies anything and the uses were noncommercial. Neither of those applies to the AI scrapers - they’re aggressively overusing freely-provided services (listen to some of the other folks on this thread about how the scrapers behave) and they’re doing so to create a competing commercial products. I’m not arguing you shouldn’t be annoyed by these chang…
I'd like to focus on your strongest point, which is the cost to the companies. I would love to know what that increase in cost looks like. You can install nginx on a tiny server and serve 10k rps of static content, or like 50 (not 50k) rps of a random web framework that generates the same content. So this increase in cost must be weighed against how efficient the software serving that content is.
If this Github post included a bunch of numbers and details demonstrating how they have reached the end of the line on optimizing their web frontend, they have ran out of things to cache, and the increase in costs is a real cause for concern to the company (not just a quick shave to the bottom line, not a bigger net/compute check written from Github to their owners), I'd throw my hands up with them and start rallying against the (unquestionably inefficient and on the line of hostile) AI agent scrapers causing the increase in traffic.
Because they did not provide that information, I have to assume that Github and Microsoft are doing this out of pure profit motivations and have abandoned any sense of commitment to open access of software. In fact, they have much to gain from building the walls of their garden up as high as they can get away with, and I'm skeptical their increase in costs is very material at all.
I would rather support services that don't camouflage as open and free software proponents one day and victims of a robbery on the next. I still think this question is important and valid: There is tons of software on Github written by users who wish for their work to remain open access. Is that the class of software and people you believe should be shuffled around into smaller and smaller services that haven't yet abandoned the commitments that allowed them to become popular?
Are the scraper sites using a large number of IP addresses, like a distributed denial of service attack? If not, rather than explicit blocking, consider using fair queuing. Do all the requests from IP addresses that have zero requests pending. Then those from IP addresses with one request pending, and so forth. Each IP address contends with itself, so making massive numbers of requests from one address won't cause a…
Yes, LLM-era scrapers are frequently making use of large numbers of IP addresses from all over the place. Some of them seem to be bot nets, but based on IP subnet ownership it seems also pretty frequently to be cloud companies, many of them outside the US. In addition to fanning out to different IPs, many of the scrapers appear to use User Agent strings that are randomised, or perhaps in some cases themselves generat…
Earlier quoted context omitted.
Because our sites are written in layers of abstraction and terrible design, which leads to requests taking serious server resources. If we hosted everything "well", you'd get a few 10-20k req/s per CPU core, but we aren't.
True. I am simply wondering -- is the resistance from executives' so powerful that it can be never overpowered? Can't we ever just tell them "Look, this is like your car with plastic suspension -- it will work for a few days or even months but we can't rely on it forever; it's time to do it proper"? Especially when the car's plastic suspension is costing them extra money? I don't get it here, for real. I would think…
However, execs I know lease cars, not buy them, for that exact reason. You don't care if the suspension is made of plastic, if it's a subscription model. The metaphor very much falls apart but I had a point somewhere.
Earlier quoted context omitted.
Huh, that sounds very reasonable, and it's the first time I've heard it mentioned. Why isn't this more wide-spread?
Complex, stateful. I'm not even sure what that would look like for a huge service like GitHub. Where do you hold those many thousands of concurrent http connections and their pending request queues in a way that you can make decisions on them while making more operational sense than a simple rate limit? A lot of things would be easy if it were viable to have one big all-knowing giga load balancer. I remember Rap Geni…
Holding open an idle HTTP connection is cheap today. That's the use case for "async". Servicing a Github fetch is much more expensive.