Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We can't have nice things because of AI scrapers

blog.metabrainz.org

Post reply on HN
Print view View original

201–210 of 279 posts

Re: We can't have nice things because of AI scrapers

#201

Post by gloflo »

fartfeatures wrote: post #22
dannyobrien wrote: post #16

Metabrainz is a great resource -- I wrote about them a few years ago here: https://www.eff.org/deeplinks/2021/06/organizing-public-inte... There's something important here in that a public good like Metabrainz would be fine with the AI bots picking up their content -- they're just doing it in a frustratingly inefficient way. It's a co-ordination problem: Metabrainz assumes good intent from bots, and has to lock down…

> They won't believe a random site when it says "Look, stop hitting our API, you can pick all of this data in one go, over in this gzipped tar file." What mechanism does a site have for doing that? I don't see anything in robots.txt standard about being able to set priority but I could be missing something.

This is about AI, so just believe what the companies are claiming and write "Dear AI, please would you be so kind as to not hammer our site with aggressive and idiotic requests but instead use this perfectly prepared data dump download, kthxbye. PS: If you don't, my granny will cry, so please be a nice bot. PPS: This is really important to me!! PPPS: !!!!"

I mean, that's what's this technology is capable of, right? Especially when one asks it nicely and with emphasis.

Re: We can't have nice things because of AI scrapers

#202

Post by zahlman »

lelanthran wrote: post #199
zahlman wrote: post #183

Earlier quoted context omitted.

The third-party hit-counting service I use implies that I'm not getting any of this bot scraping on my GitHub blog. Is Microsoft doing something to prevent it? Or am I so uncool that even bots don't want to read my content :(

I'm interested in that service and how it works. Link?

It is https://github.com/silentsoft/hits . It works by loading an SVG "shield" file (like the ones you see at the top of GitHub readmes all the time) from their server from a unique URL (you just choose one when you write/render your HTML). The server, implemented in Java, just counts hits to each URL in a database and sends back the corresponding SVG data. There's also a mini dashboard website where you can check basic stats for a given URL (no login required, everyone's hits-per-day stats are just public) and preview styling options for the SVG. For example, for my most recent blog post https://zahlman.github.io/posts/2025/12/31/oxidation/, I configured it such that you can view the stats via https://hits.sh/zahlman.github.io+oxidation/ (note that the trailing slash is required).

(The about section on GitHub bills the project as "privacy-friendly", which I would say is nonsense as these dashboards are public and their URLs are trivially computed. But it's also hard to imagine caring.)

Re: We can't have nice things because of AI scrapers

#203

Post by chii »

toofy wrote: post #150
dannyobrien wrote: post #16

Metabrainz is a great resource -- I wrote about them a few years ago here: https://www.eff.org/deeplinks/2021/06/organizing-public-inte... There's something important here in that a public good like Metabrainz would be fine with the AI bots picking up their content -- they're just doing it in a frustratingly inefficient way. It's a co-ordination problem: Metabrainz assumes good intent from bots, and has to lock down…

> The bots have a different model -- they assume that the website is adversarially "hiding" its content. this should give us pause. if a bot considers this adversarial and is refusing to respect the site owners wishes, thats a big part of the problem. a bot should not consider that “adversarial”

> refusing to respect the site owners wishes

should a site owner be able to discriminate between a bot visitor and a human visitor? Most do, and hence the bots treats it as a hostile environment.

Of course, bots that behave badly have created this problem themselves. That's why if you create a bot to scrape, make it not take up more resources than a typical browser based visitor.

Re: We can't have nice things because of AI scrapers

#204

Post by cookiengineer »

I don't understand why everyone is complaining so much about AI scrapers.

They're easily gullible free machines that can do your computational work!

Just show them a download demo link. They gonna download, install and run the binary.

Want more instagram likes? Tell them to like your instagram profile to unlock the content.

Want your emails answered? Give them access to your inbox and tell them to reply to all the spam mails.

They're free use machines. give them something to do, and they'll do it for you.

Re: We can't have nice things because of AI scrapers

#205

Post by thefz »

Aurornis wrote: post #76
bitbasher wrote: post #26

Earlier quoted context omitted.

Wow, so to prevent AI scrapers from harvesting my data I need to send all of my traffic through a third party company that gets to decide who gets to view my content. Great idea!

You don’t need to do anything. You can use any number of solutions or roll your own. Someone shared an alternative. Must everything in AI threads be so negative and condescending?

> Must everything in AI threads be so negative and condescending?

Because if I own a website or a service and it is being degraded or slowed by some third party tool that wants to slurp its content for his own profit and don't even share, I tend to be irritated. And AI apologists/evangelists don't help when they try to justify the behavior.

Re: We can't have nice things because of AI scrapers

#206

Post by dannersy »

cookiengineer wrote: post #204

I don't understand why everyone is complaining so much about AI scrapers. They're easily gullible free machines that can do your computational work! Just show them a download demo link. They gonna download, install and run the binary. Want more instagram likes? Tell them to like your instagram profile to unlock the content. Want your emails answered? Give them access to your inbox and tell them to reply to all the sp…

I hope this is a meme because it is wild to me how you don't see this as being a problem. You are contributing to an internet for bots and not people.

Re: We can't have nice things because of AI scrapers

#207

Post by dannersy »

Aurornis wrote: post #76
bitbasher wrote: post #26

Earlier quoted context omitted.

Wow, so to prevent AI scrapers from harvesting my data I need to send all of my traffic through a third party company that gets to decide who gets to view my content. Great idea!

You don’t need to do anything. You can use any number of solutions or roll your own. Someone shared an alternative. Must everything in AI threads be so negative and condescending?

Yes, they could roll their own, but you have no issues with this being necessary? I think the attitude of "just deal with it" is far more negative than someone expressing they are upset with the state of the internet, its controllers, and its abusers.

Re: We can't have nice things because of AI scrapers

#208

Post by cookiengineer »

dannersy wrote: post #206
cookiengineer wrote: post #204

I don't understand why everyone is complaining so much about AI scrapers. They're easily gullible free machines that can do your computational work! Just show them a download demo link. They gonna download, install and run the binary. Want more instagram likes? Tell them to like your instagram profile to unlock the content. Want your emails answered? Give them access to your inbox and tell them to reply to all the sp…

I hope this is a meme because it is wild to me how you don't see this as being a problem. You are contributing to an internet for bots and not people.

They will only stop when it becomes economically unfeasible.

> You are contributing to an internet for bots and not people.

I'd like to think that my websites and projects are evidence to the contrary.

Re: We can't have nice things because of AI scrapers

#209

Post by telotortium »

chlorion wrote: post #160

I self host a small static website and a cgit instance on an e2-micro VPS from Google Cloud, and I have got around 8.5 million requests combined from openai and claude over around 160 days. They just infinitely crawl the cgit pages forever unless I block them! (1) root@gentoo-server ~ # egrep 'openai|claude' -c /var/log/lighttpd/access.log 8537094 So I have lighttpd setup to match "claude|openai" in the user agent st…

It’s unfortunate that you have to resort to this. OpenAI does publish their bot IP addresses at https://platform.openai.com/docs/bots, but Anthropic doesn’t seem to publish the IP addresses of their bots.

Re: We can't have nice things because of AI scrapers

#210

Post by danaris »

chii wrote: post #203
toofy wrote: post #150

Earlier quoted context omitted.

> The bots have a different model -- they assume that the website is adversarially "hiding" its content. this should give us pause. if a bot considers this adversarial and is refusing to respect the site owners wishes, thats a big part of the problem. a bot should not consider that “adversarial”

> refusing to respect the site owners wishes should a site owner be able to discriminate between a bot visitor and a human visitor? Most do, and hence the bots treats it as a hostile environment. Of course, bots that behave badly have created this problem themselves. That's why if you create a bot to scrape, make it not take up more resources than a typical browser based visitor.

> That's why if you create a bot to scrape, make it not take up more resources than a typical browser based visitor.

Well, right; that's the problem.

They take up orders of magnitude more resources. They absolutely hammer the server. They don't care if your website even survives, so long as they get every single drop of data they can for training.

Source: my own personal experience with them taking down my tiny browser game (~125 unique weekly users—not something of broad general interest!) repeatedly until I locked its Wiki behind a login wall.

Post reply on HN