An update on residential proxies and the scraper situation
1–10 of 422 posts
Re: An update on residential proxies and the scraper situation
#2mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
Re: An update on residential proxies and the scraper situation
#3The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
Re: An update on residential proxies and the scraper situation
#4There is a large community of people that poison scrapers.
The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc.
Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...
Re: An update on residential proxies and the scraper situation
#5I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
Re: An update on residential proxies and the scraper situation
#6I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.
Re: An update on residential proxies and the scraper situation
#7There is a large community of people that poison scrapers. The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc. Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...
People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue.
The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.
Re: An update on residential proxies and the scraper situation
#8The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
Re: An update on residential proxies and the scraper situation
#9I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
Re: An update on residential proxies and the scraper situation
#10Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.