An update on residential proxies and the scraper situation • Hacker News Board

Re: An update on residential proxies and the scraper situation

#121

Post by dredmorbius » Sat, Jul 11, 2026, 4:21 AM UTC

dang wrote: post #70

dredmorbius wrote: post #32

Earlier quoted context omitted.

Seriously, dang? 10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned. This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose acti…

The most recent 60 (!) comments plus every submission of the last 6 months were all about the same thing. That's extreme. The posts didn't all mention that specific project, but there was only one topic and they were extremely repetitive. This is not a close call. I made it all the way back to https://news.ycombinator.com/posts?id=atomic128&next=4628060... (6 months ago) before seeing posts about anything else, only…

Exceptions?

https://news.ycombinator.com/item?id=47116093> (LLM but not PF).

https://news.ycombinator.com/item?id=47095664> (a16h)

https://news.ycombinator.com/item?id=46695693> (vuln exploits) 2026-1-20

https://news.ycombinator.com/item?id=46280602> (???, but not PF)

https://news.ycombinator.com/item?id=46195234> (Monero / Dark Web) 2025-12-8

https://news.ycombinator.com/item?id=45894305> and https://news.ycombinator.com/item?id=45826273> (Tor hidden service) Nov 2025

That's from the past 20 comments.

Submissions: 8 most recent on PF, 9+ cover nuclear power, Tor dark web, robotaxis, and other topics.

Again: Not a one-note flute, though fairly focused of late on AI and poisoning.

Again: I think the ban is unwarranted. I'm not sure what's driving your thinking here, but a no-warnings ban seems excessive. And given YC's current preponderance of AI/agentic launches (https://news.ycombinator.com/launches>), self-serving and contrary to the "we moderate YC stories less" guideline.

(Yes, I'm aware "less" isn't "none", and this is an account/user rather than story, I hope my point stands and is clear.)

The Yann LeCun posts you link are ... a bit OTT. That's also a couple of years ago.

I've said my bit. I'm hoping you and atomic128 can come to an understanding in email.

Re: An update on residential proxies and the scraper situation

#122

Post by teravor » Sat, Jul 11, 2026, 4:31 AM UTC

I find the notion that you would use residential proxies to scrape LWN somewhat laughable, I'm reading this article using a VPN.

residential proxy bandwidth isn't that cheap, I could see it be used on a reddit (though i would probably just mass register accounts to bypass their block instead).

Re: An update on residential proxies and the scraper situation

#123

Post by ezst » Sat, Jul 11, 2026, 4:31 AM UTC

cyberax wrote: post #93

andai wrote: post #45

Earlier quoted context omitted.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…

Hah. I have a homelab with a couple of sites, including a personal Forgejo installation. Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough. However, this time they hammered my computer for 12 hours with about 200 requests per…

The paradox of them selling "intelligence on demand" or "coding agents rivaling the best developers" and yet having dumb as fart bots/scrapers is lost on many. But not all.

Re: An update on residential proxies and the scraper situation

#124

Post by teravor » Sat, Jul 11, 2026, 4:35 AM UTC

dang wrote: post #17

atomic128 wrote: post #4

There is a large community of people that poison scrapers. The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc. Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...

I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now. We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for. https://news.ycombinator.com/newsguidelines.html Edit: If you don't want to be banned, you're…

I'm confused why everyone is pretending that a ban holds any meaning here. he probably already has a new account, it might have made sense to just silently ban him in the hope of imposing a minor cost on what you believe he is doing, but wasting your time addressing it imposed a far greater cost on you than him.

Re: An update on residential proxies and the scraper situation

#125

Post by dang » Sat, Jul 11, 2026, 4:39 AM UTC

khurs wrote: post #115

dang wrote: post #104

Earlier quoted context omitted.

It was, but there were too many legit users getting affected, so we turned most of that off a few days ago. Are you still seeing it?

Not in last day no.

Ok that's good - if you get it again can you let us know at hn@ycombinator.com? We definitely don't want to exclude legit users.

Re: An update on residential proxies and the scraper situation

#126

Post by oasisbob » Sat, Jul 11, 2026, 4:41 AM UTC

dang wrote: post #69

fragmede wrote: post #42

Earlier quoted context omitted.

How does HN fare with scraper load? Is it just CDN and pay the extra bandwidth bill for anon hit requests?

It's really bad. I found myself identifying with everything Jonathan wrote in the OP - so much so that I thought of asking to compare notes on mitigation measures.

As someone dealing with similar on a large site, I'd love to see a private community to discuss some of these issues.

Re: An update on residential proxies and the scraper situation

#127

Post by noncoml » Sat, Jul 11, 2026, 4:44 AM UTC

harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

Or you can go full Reddit and just block anything that seems even remotely suspicious.

Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too.

Using VPN? Blocked.

Your iPhone is too old, blocked.

Your screen brightness too low? Believe or not, blocked.

Re: An update on residential proxies and the scraper situation

#128

Post by ammario » Sat, Jul 11, 2026, 4:47 AM UTC

PlasmaPower wrote: post #107

harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck. Then again, a large portion of the problem seems to be bots making way too many requests and in general not being…

There are PoW approaches that even the playing field between data centers and desktops. RandomX is my favorite.

Re: An update on residential proxies and the scraper situation

#129

Post by [deleted] » Sat, Jul 11, 2026, 4:54 AM UTC

BoorishBears wrote: post #94

zuzululu wrote: post #75

Earlier quoted context omitted.

ive seen unscrupulous founders fake traction during diligence, which is my day job but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection

[flagged]

[deleted]

Re: An update on residential proxies and the scraper situation

#130

Post by BoorishBears » Sat, Jul 11, 2026, 4:55 AM UTC

llbbdd wrote: post #119

BoorishBears wrote: post #94

Earlier quoted context omitted.

[flagged]

> forced lowercase ??? shift is an extra key to press

https://www.businessinsider.com/lowercase-typing-altman-dors...

most shibboleths are subtle like that