Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

61–70 of 422 posts

Re: An update on residential proxies and the scraper situation

#61

Post by harshreality »

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.

It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.

The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.

For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a large improvement over a site-hosted captcha. I would rather, and I think most humans would agree, have to wait a few seconds, maybe even closer to a minute in the future, to get a website access token good for a day or a week, than be forced to solve a captcha.

The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.

[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.

Re: An update on residential proxies and the scraper situation

#62

Post by noxvilleza »

TurdF3rguson wrote: post #49
noxvilleza wrote: post #26

Earlier quoted context omitted.

Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential…

Yes I've seen it. ClaudeBot will gleefully announce itself when it hammers my niche website a million times a day.

At least those bots are easy to block though. I run a niche stats website for an esport and I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data - most of what they scrape is directly available from Valve's APIs.

Re: An update on residential proxies and the scraper situation

#63

Post by fragmede »

dredmorbius wrote: post #53
fragmede wrote: post #42

Earlier quoted context omitted.

How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?

For one datapoint ... I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...). Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given tha…

HN is exported to firebase, which you can hit directly, for that sort of purpose

https://github.com/HackerNews/API

Re: An update on residential proxies and the scraper situation

#64

Post by dredmorbius »

fragmede wrote: post #63
dredmorbius wrote: post #53

Earlier quoted context omitted.

For one datapoint ... I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...). Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given tha…

HN is exported to firebase, which you can hit directly, for that sort of purpose https://github.com/HackerNews/API

I know that.

I've not worked with the API, and there's the blessing/curse (blurse‽) that HTML is a known, if poor, standard.

API always translates to "one more thing to learn, that's applicable to a single-use case". HTML scraping / sorting I can apply across multiple sites.

That said, a standard, say, JSON packaging of website contents available on request might be fun to have.

Re: An update on residential proxies and the scraper situation

#65

Post by BoorishBears »

Bratmon wrote: post #59
BoorishBears wrote: post #24

Earlier quoted context omitted.

Thank god for residential proxies. Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet

By providing a way for corporate AI scrapers to operate with impunity and force the last few independently-run websites to move to the cloud?

No one's firing up a residential proxy to read your blog, and the corporate AI scrapers have all the resources in the world even without residential proxies

They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.

Re: An update on residential proxies and the scraper situation

#66

Post by TurdF3rguson »

noxvilleza wrote: post #62
TurdF3rguson wrote: post #49

Earlier quoted context omitted.

Yes I've seen it. ClaudeBot will gleefully announce itself when it hammers my niche website a million times a day.

At least those bots are easy to block though. I run a niche stats website for an esport and I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data - most of what they scrape is directly available from Valve's APIs.

> I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data

Probably as simple as the fact that there are unmetered residential proxy plans, which means once you're already paying for one, there's no reason not to use it for everything.

Re: An update on residential proxies and the scraper situation

#68

Post by mips_avatar »

andai wrote: post #45
mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…

I kind of wish the recent Google monopoly court ruling had forced Google to open up their index to anyone, not just Perplexity/other big players.

Re: An update on residential proxies and the scraper situation

#69

Post by dang »

fragmede wrote: post #42
dang wrote: post #31

One article mentioned in the OP was discussed here: Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)

How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?

It's really bad. I found myself identifying with everything Jonathan wrote in the OP - so much so that I thought of asking to compare notes on mitigation measures.

Re: An update on residential proxies and the scraper situation

#70

Post by dang »

dredmorbius wrote: post #32
dang wrote: post #17

Earlier quoted context omitted.

I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now. We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for. https://news.ycombinator.com/newsguidelines.html Edit: If you don't want to be banned, you're…

Seriously, dang? 10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned. This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose acti…

The most recent 60 (!) comments plus every submission of the last 6 months were all about the same thing. That's extreme. The posts didn't all mention that specific project, but there was only one topic and they were extremely repetitive. This is not a close call.

I made it all the way back to https://news.ycombinator.com/posts?id=atomic128&next=4628060... (6 months ago) before seeing posts about anything else, only to find that there was a different agenda before that. Not cool.

Edit: and before all that, there was this: https://news.ycombinator.com/posts?id=atomic128&next=4164795.... This is obviously not using HN as intended.

Post reply on HN