We can't have nice things because of AI scrapers • Hacker News Board

Re: We can't have nice things because of AI scrapers

#171

Post by janc_ » Wed, Jan 14, 2026, 2:57 AM UTC

jmward01 wrote: post #94

Bummer. I have used them a lot when I was ripping my cds. Anonymity is a massive value of the web (at least the appearance of anonymity). I wonder if there is a way to have a central anonymous system that just relays trust, not identity. So maybe something like you can get a token but its trust is very nearly zero until you combine it with other tokens. Combining tokens combines their trust and their consequences. If…

These AI crawlers already steal residential user connections to do their scraping. They'll happily steal your trust tokens too…

Re: We can't have nice things because of AI scrapers

#172

Post by adrianwaj » Wed, Jan 14, 2026, 3:02 AM UTC

Look no further than x402 micropayments as both the solution and opportunity here.

And then a way to return a portion to humans.

These AI companies are loaded too (maybe not the long-tail as yet) and the crypto ecosystem is mature.

Come one, come all. Make money.

Need a Wordpress plugin to start the ball rolling and provide ping endpoints for the AI companies to leach from. They can pay to get those pings too.

Give them what they want and charge them. Lower their costs by making their scraping more efficient.

Re: We can't have nice things because of AI scrapers

#173

Post by zzzeek » Wed, Jan 14, 2026, 3:05 AM UTC

godelski wrote: post #139

zzzeek wrote: post #47

Earlier quoted context omitted.

Citation needed

One of the most popular ones is Anubis. It uses a proof of work and can even do poisoning: https://anubis.techaro.lol/ They even mention iocaine. I know, inconceivable!: https://iocaine.madhouse-project.org/ There's also tons of HN posts on the topic with varying solutions: https://news.ycombinator.com/item?id=45935729 https://news.ycombinator.com/item?id=45711094 https://news.ycombinator.com/item?id=44142761 https:/…

Anubis is the only tool that claims to have heuristics to identify a bot, but my understanding is that it does this by presenting obnoxious challenges to all users. Not really feasible. Old school approaches like ip blocking or even ASN blocking are obsolete - these crawlers purposely spam from thousands of IPs, and if you block them on a common ASN, they come back a few days later from thousands of unique ASNs. So this is not really a "roll your own" situation, especially if you are running off the shelf software that doesn't have some straightforward means of building in these various approaches of endless page mazes (which I would still have to serve anyway).

Re: We can't have nice things because of AI scrapers

#174

Post by Cadwhisker » Wed, Jan 14, 2026, 3:06 AM UTC

tux wrote: post #103

dannyobrien wrote: post #16

Metabrainz is a great resource -- I wrote about them a few years ago here: https://www.eff.org/deeplinks/2021/06/organizing-public-inte... There's something important here in that a public good like Metabrainz would be fine with the AI bots picking up their content -- they're just doing it in a frustratingly inefficient way. It's a co-ordination problem: Metabrainz assumes good intent from bots, and has to lock down…

Yeah AI scrapers is one of the reasons why i have closed my public website https://tvnfo.com and only left donors site online. It’s not only because of AI scrapers but i grew tired of people trying to scrape the site eating a lot of reasorcers this small project don’t have. Very sad really it was publicly online since 2016. Now it’s only available for donors. Running a tiny project on just $60 a month. If this was no…

I took a look at the https://tvnfo.com/ site and I have no idea what's behind the donation wall. Can I suggest you have a single page which explains or demonstrates the content, or there's no reason for "new" people to want to donate to get access.

Re: We can't have nice things because of AI scrapers

#175

Post by wtetzner » Wed, Jan 14, 2026, 3:09 AM UTC

what wrote: post #155

squigz wrote: post #38

Earlier quoted context omitted.

> To archive Metabrainz there is no way but to browse the pages slowly page-by-page. There's no machine-communicable way that suggests an alternative. Why does there have to be a "machine-communicable way"? If these developers cared about such things they would spend 20 seconds looking at this page. It's literally one of the first links when you Google "metabrainz" https://metabrainz.org/datasets

You expect the developers of a crawler to look at every site they crawl and develop a specialized crawler for them? That’s fine if you’re only crawling a handful of sites, but absolutely insane if you’re crawling the entire web.

Isn't the point of AI that it's good at understanding content written for humans? Why can't the scrapers run the homepage through an LLM to detect that?

I'm also not sure why we should be prioritizing the needs of scraper writers over human users and site operators.

Re: We can't have nice things because of AI scrapers

#176

Post by wtetzner » Wed, Jan 14, 2026, 3:12 AM UTC

TeMPOraL wrote: post #131

VonGallifrey wrote: post #118

Earlier quoted context omitted.

> turns out plenty of people say they write for others to read LLMs are not people. They don't write blogs so that a company can profit from their writing by training LLMs on it. They write for others to read their ideas.

LLMs aren't making their owners money by just idling on datacenters worth of GPU. They're making money by being useful for users that pay for access . The knowledge and insights from writings that go into training data all end up being read by people directly, as well as inform even more useful output and work benefiting even more people.

Except the output coming from an LLM is the LLM's take on it, not the original source material. It's not the same thing. Not all writing is simply a collection of facts.

Re: We can't have nice things because of AI scrapers

#177

Post by username223 » Wed, Jan 14, 2026, 3:34 AM UTC

TeMPOraL wrote: post #95

tooltower wrote: post #80

> Rather than downloading our dataset in one complete download, they insist on loading all of MusicBrainz one page at a time. Is there a standard mechanism for batch-downloading a public site? I'm not too familiar with crawlers these days.

There isn't. There never was one, because vast majority of websites are actually selfish with respect to data, even when that's entirely pointless. You can see this even here, with how some people complain LLMs made them stop writing their blogs: turns out plenty of people say they write for others to read, but they care more about tracking and controlling the audience. Anyway, all that means there was never a critic…

> turns out plenty of people say they write for others to read, but they care more about tracking and controlling the audience.

I couldn't care less about "tracking and controlling the audience," but I have no interest in others using my words and photos to profit from slop generators. I make that clear in robots.txt and licenses, but they ignore both.

Re: We can't have nice things because of AI scrapers

#178

Post by phyzome » Wed, Jan 14, 2026, 3:35 AM UTC

garganzol wrote: post #152

phyzome wrote: post #109

Earlier quoted context omitted.

We already know the solution: One well-behaved, shared scraper could serve all of the AI companies simultaneously. The problem is that they're not doing it.

This is an interesting approach. Archive.org could be such a solution, kind of. Not its cold storage as it's now, but a warm access layer. Sponsorship by AI companies would a good initiative for the project.

I can't imagine IA ever going for it. You'd need a separate org that just scrapes for AI training, because its bot is going to be blocked by anyone who is anti-AI. It wouldn't make sense for it to serve multiple purposes.

Common Crawl would be a better fit, but still might not want to serve in that capacity.

Re: We can't have nice things because of AI scrapers

#179

Post by dang » Wed, Jan 14, 2026, 3:47 AM UTC

chlorion wrote: post #160

I self host a small static website and a cgit instance on an e2-micro VPS from Google Cloud, and I have got around 8.5 million requests combined from openai and claude over around 160 days. They just infinitely crawl the cgit pages forever unless I block them! (1) root@gentoo-server ~ # egrep 'openai|claude' -c /var/log/lighttpd/access.log 8537094 So I have lighttpd setup to match "claude|openai" in the user agent st…

And those are the good actors! We're under a crawlocalpyse from botnets, er, residential proxies.

"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", anyone?

Re: We can't have nice things because of AI scrapers

#180

Post by tux » Wed, Jan 14, 2026, 3:51 AM UTC

Cadwhisker wrote: post #174

tux wrote: post #103

Earlier quoted context omitted.

Yeah AI scrapers is one of the reasons why i have closed my public website https://tvnfo.com and only left donors site online. It’s not only because of AI scrapers but i grew tired of people trying to scrape the site eating a lot of reasorcers this small project don’t have. Very sad really it was publicly online since 2016. Now it’s only available for donors. Running a tiny project on just $60 a month. If this was no…

I took a look at the https://tvnfo.com/ site and I have no idea what's behind the donation wall. Can I suggest you have a single page which explains or demonstrates the content, or there's no reason for "new" people to want to donate to get access.

Yeah i’ll have something up soon :-)