Live data from Hacker News

We can't have nice things because of AI scrapers

blog.metabrainz.org

131–140 of 279 posts

Re: We can't have nice things because of AI scrapers

#131

Earlier quoted context omitted.

There isn't. There never was one, because vast majority of websites are actually selfish with respect to data, even when that's entirely pointless. You can see this even here, with how some people complain LLMs made them stop writing their blogs: turns out plenty of people say they write for others to read, but they care more about tracking and controlling the audience. Anyway, all that means there was never a critic…

> turns out plenty of people say they write for others to read LLMs are not people. They don't write blogs so that a company can profit from their writing by training LLMs on it. They write for others to read their ideas.

LLMs aren't making their owners money by just idling on datacenters worth of GPU. They're making money by being useful for users that pay for access. The knowledge and insights from writings that go into training data all end up being read by people directly, as well as inform even more useful output and work benefiting even more people.

Re: We can't have nice things because of AI scrapers

#132
post #79

Earlier quoted context omitted.

I'm not entirely sure why people think more standards are the way forward. The scrapers apparently don't listen to the already-established standards. What makes one think they would suddenly start if we add another one or two?

There is no standard, well-known way for a website to advertise, "hey, here's a cached data dump for bulk download, please use that instead of bulk scraping". If they were, I'd expect the major AI companies and other users[0] to use that method for gathering training data[1]. They have compelling reasons to: it's cheaper for them, and cultivates goodwill instead of burning it. This also means that right now, it could…

You do realize that these AI scrapers are most likely written by people who have no idea what they're doing right? Or they just don't care? If they were, pretty much none of the problems these things have caused would exist. Even if we did standardize such a thing, I doubt they would follow it. After all, they think they and everyone else has infinite resources so they can just hammer websites forever.

Re: We can't have nice things because of AI scrapers

#133
post #78
post #57

The SQLite team faced a similar problem last year, and Richard Hipp (the creator of SQLite) made almost the same comment: "The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things...." https://sqlite.org/…

[flagged]

"Why don't you just clone the repo?" Yes. Why dont you?

If you're gonna grab a repo to make a code theft machine then at least dont ddos the servers while you're at it.

Re: We can't have nice things because of AI scrapers

#134
post #116

Earlier quoted context omitted.

I use iocaine[0] to generate a tarpit. Yesterday it served ~278k "pages" consisting of ~500MB of gibberish (and that's despite banning most AI scrapers in robots.txt.) [0] https://iocaine.madhouse-project.org

Can't seem to access this. It flashes some text briefly then gives me an 418 TEAPOT response. I wonder if it's because I'm on Linux? EDIT: Begrudgingly checked Chrome, and it loads. I guess it doesn't like Firefox?

Doesn't work on my firefox either.

Friendly fire, I suppose.

Re: We can't have nice things because of AI scrapers

#135
post #44

Earlier quoted context omitted.

> In fact firefox now allows you to preview the link and get key points without ever going to the link[1] > [1] https://imgur.com/a/3E17Dts This is generated on device with llama.cpp compiled to webassembly (aka wllama) and running SmolLM2-360M. [1] How is this different from the user clicking on the link? In the end, your local firefox will fetch the link in order to summarize it, the same way you would have followe…

I actually didnt know it was using a local model and that it fetches it locally.

They should advertise that. I pretty much reflexively avoid any mention of AI in interfaces because they usually mean "we're sending this all to openthropigoogosoft so I hope you don't have any secrets."

Re: We can't have nice things because of AI scrapers

#136
post #78

Earlier quoted context omitted.

[flagged]

"Why don't you just clone the repo?" Yes. Why dont you? If you're gonna grab a repo to make a code theft machine then at least dont ddos the servers while you're at it.

[flagged]

Re: We can't have nice things because of AI scrapers

#137
post #132

Earlier quoted context omitted.

There is no standard, well-known way for a website to advertise, "hey, here's a cached data dump for bulk download, please use that instead of bulk scraping". If they were, I'd expect the major AI companies and other users[0] to use that method for gathering training data[1]. They have compelling reasons to: it's cheaper for them, and cultivates goodwill instead of burning it. This also means that right now, it could…

You do realize that these AI scrapers are most likely written by people who have no idea what they're doing right? Or they just don't care? If they were, pretty much none of the problems these things have caused would exist. Even if we did standardize such a thing, I doubt they would follow it. After all, they think they and everyone else has infinite resources so they can just hammer websites forever.

I realise you are making assertions for which you have no evidence. Until a standard exists we can't just assume nobody will use it, particularly when it makes the very task they are scraping for simpler and more efficient.

Re: We can't have nice things because of AI scrapers

#138
AI is destroying the free internet along with everything else

My web host suspended my website account last week due to a sudden large volume of requests to it - effectively punishing me for being scraped by bots.

I've had to move to a new host to get back up, but what hope does the little guy have? it's like GPU and ram prices, it doesn't matter if I pay 10x 100x or 1000x more than I did, the AI companies have infinite resources, and they don't care what damage they do in the rush to become the no 1 in the industry

The cynic in me would say it's intentional, destroy all the free sites so you have to get your info from their ai models, price home users out of high end hardware so they have to lease the functions from big companies

Re: We can't have nice things because of AI scrapers

#139
post #47

Earlier quoted context omitted.

You can implement this yourself, who is stopping you?

Citation needed

One of the most popular ones is Anubis. It uses a proof of work and can even do poisoning: https://anubis.techaro.lol/

They even mention iocaine. I know, inconceivable!: https://iocaine.madhouse-project.org/

There's also tons of HN posts on the topic with varying solutions:

https://news.ycombinator.com/item?id=45935729

https://news.ycombinator.com/item?id=45711094

https://news.ycombinator.com/item?id=44142761

https://news.ycombinator.com/item?id=44378127

Re: We can't have nice things because of AI scrapers

#140
post #134
post #116

Earlier quoted context omitted.

Can't seem to access this. It flashes some text briefly then gives me an 418 TEAPOT response. I wonder if it's because I'm on Linux? EDIT: Begrudgingly checked Chrome, and it loads. I guess it doesn't like Firefox?

Doesn't work on my firefox either. Friendly fire, I suppose.

Works on my Firefox. Mac and Linux
Post reply on HN