An update on residential proxies and the scraper situation

Post by dgacmu » Sat, Jul 11, 2026, 12:46 PM UTC

andai wrote: post #45

mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…

Post by hananova » Sat, Jul 11, 2026, 1:14 PM UTC

inigyou wrote: post #223

TZubiri wrote: post #78

Earlier quoted context omitted.

Why the air quotes? Evading a ban and using (potentially ill gotten) residential ips to circumvent that refusal of service, is a bad actor.

Surely that depends on the motivation for the ban.

Post by robinsonb5 » Sat, Jul 11, 2026, 1:28 PM UTC

inigyou wrote: post #226

aorth wrote: post #178

Earlier quoted context omitted.

The excessive scraping and ignoring robots.txt only breaks the informal social contract established over the past decades of the open internet. The real problem is the companies offering money to developers if they include unrelated SDKs in their calculator or flashlight (for example) applications. Those SDKs add functionality to incorporate those devices into a network that can be used for scraping. The traffic is l…

On what grounds would it be illegal though? Things don't become illegal just because you don't like them. They may become illegal just because the president doesn't like them, but I don't think you're him, and in the absence of that, there has to be a majority of Congress and most of them want a reason.

Post by herpdyderp » Sat, Jul 11, 2026, 1:36 PM UTC

inigyou wrote: post #244

RickHull wrote: post #240

Earlier quoted context omitted.

This seems nonsensical. Care to elaborate?

If Google determines you're an undesirable user, doing the captcha is just an exercise to waste your time.

Post by flaburgan » Sat, Jul 11, 2026, 2:04 PM UTC

jay_kyburz wrote: post #13

mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape. You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.