Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

41–50 of 422 posts

Re: An update on residential proxies and the scraper situation

#41

Post by andai »

>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.

I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.

Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?

Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)

Re: An update on residential proxies and the scraper situation

#42

Post by fragmede »

dang wrote: post #31

One article mentioned in the OP was discussed here: Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)

How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?

Re: An update on residential proxies and the scraper situation

#43

Post by mschuster91 »

cyanydeez wrote: post #2

mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS. The question is more about why the US and others can't properly enforce the bullshit all this amounts to.

> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.

It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.

Re: An update on residential proxies and the scraper situation

#44

Post by fragmede »

eduction wrote: post #16

Can BitTorrent’s architecture contribute anything useful here? I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.

Yes but it's getting bot owners to use it is the problem. There's already the common crawl repository to start with but it isn't being used.

Re: An update on residential proxies and the scraper situation

#45

Post by andai »

mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?

Edit: the article says millions of times per hour? (!?)

The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?

Re: An update on residential proxies and the scraper situation

#46

Post by zb3 »

> widespread scraping of web sites in search of training data for large language models and related projects

This is a good thing, thanks to this we have powerful open source LLMs.

> This activity overwhelms sites with traffic.

When LLMs get good enough, we won't need those sites anymore :)

[not satire, this is what I think, without self-censorship]

Re: An update on residential proxies and the scraper situation

#47

Post by fragmede »

dredmorbius wrote: post #32
dang wrote: post #17

Earlier quoted context omitted.

I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now. We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for. https://news.ycombinator.com/newsguidelines.html Edit: If you don't want to be banned, you're…

Seriously, dang? 10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned. This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose acti…

I think the reasoning is about having alt accounts for different purposes. He intention is to map one human to one account and have all of their thoughts from that one account, instead of one human having one account to discuss scraping on, and a different account to discuss crypto on.

Re: An update on residential proxies and the scraper situation

#48

Post by andai »

logancbrown wrote: post #7
atomic128 wrote: post #4

There is a large community of people that poison scrapers. The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc. Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...

People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue. The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.

Yeah. A fun thing to do is to try and actually read common crawl!

Really makes you think, what we're feeding them...

Re: An update on residential proxies and the scraper situation

#49

Post by TurdF3rguson »

noxvilleza wrote: post #26
everfrustrated wrote: post #6

I wonder how much of this is traffic caused by peoples agents using web tools causing searches and fetches rather than general trawls of the internet.

Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential…

Yes I've seen it. ClaudeBot will gleefully announce itself when it hammers my niche website a million times a day.

Re: An update on residential proxies and the scraper situation

#50

Post by TurdF3rguson »

Bratmon wrote: post #10

Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.

I think they also have to operate in countries that don't mind shady things like this.
Post reply on HN