An update on residential proxies and the scraper situation

Post by nekusar » Fri, Jul 10, 2026, 10:06 PM UTC

dang wrote: post #17

atomic128 wrote: post #4

There is a large community of people that poison scrapers. The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc. Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...

I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now. We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for. https://news.ycombinator.com/newsguidelines.html Edit: If you don't want to be banned, you're…

Post by jay_kyburz » Fri, Jul 10, 2026, 10:16 PM UTC

mips_avatar wrote: post #18

jay_kyburz wrote: post #15

Earlier quoted context omitted.

How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.

that's hard to do with rendered content, oftentimes the result depends on a backend service. Maybe you should make the service it's running public but that might be a line most aren't willing to cross.

Post by [deleted] » Fri, Jul 10, 2026, 10:27 PM UTC

Post by zuzululu » Fri, Jul 10, 2026, 10:32 PM UTC

logancbrown wrote: post #7

atomic128 wrote: post #4

There is a large community of people that poison scrapers. The poison gets better every day, and the community is continuously growing. Poison Fountain, alone, transmits hundreds of gigabytes of poison per day, which goes into scrapers, git repositories on every hosting platform, social media, etc. Part of the poisoning community on Reddit, for example: https://www.reddit.com/r/PoisonFountain/comments/1uocaii/a_n...

People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue. The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.

Post by sumedh » Fri, Jul 10, 2026, 10:34 PM UTC

ccgreg wrote: post #22

sixtyj wrote: post #8

The issue with scrapping is the intensity and volume of bots. I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it. Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.

A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.