Earlier quoted context omitted.
There isn't. There never was one, because vast majority of websites are actually selfish with respect to data, even when that's entirely pointless. You can see this even here, with how some people complain LLMs made them stop writing their blogs: turns out plenty of people say they write for others to read, but they care more about tracking and controlling the audience. Anyway, all that means there was never a critic…
> turns out plenty of people say they write for others to read LLMs are not people. They don't write blogs so that a company can profit from their writing by training LLMs on it. They write for others to read their ideas.
We can't have nice things because of AI scrapers
131–140 of 279 posts
Re: We can't have nice things because of AI scrapers
#132Earlier quoted context omitted.
I'm not entirely sure why people think more standards are the way forward. The scrapers apparently don't listen to the already-established standards. What makes one think they would suddenly start if we add another one or two?
There is no standard, well-known way for a website to advertise, "hey, here's a cached data dump for bulk download, please use that instead of bulk scraping". If they were, I'd expect the major AI companies and other users[0] to use that method for gathering training data[1]. They have compelling reasons to: it's cheaper for them, and cultivates goodwill instead of burning it. This also means that right now, it could…
Re: We can't have nice things because of AI scrapers
#133The SQLite team faced a similar problem last year, and Richard Hipp (the creator of SQLite) made almost the same comment: "The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things...." https://sqlite.org/…
[flagged]
If you're gonna grab a repo to make a code theft machine then at least dont ddos the servers while you're at it.
Re: We can't have nice things because of AI scrapers
#134Earlier quoted context omitted.
I use iocaine[0] to generate a tarpit. Yesterday it served ~278k "pages" consisting of ~500MB of gibberish (and that's despite banning most AI scrapers in robots.txt.) [0] https://iocaine.madhouse-project.org
Can't seem to access this. It flashes some text briefly then gives me an 418 TEAPOT response. I wonder if it's because I'm on Linux? EDIT: Begrudgingly checked Chrome, and it loads. I guess it doesn't like Firefox?
Friendly fire, I suppose.
Re: We can't have nice things because of AI scrapers
#135Earlier quoted context omitted.
> In fact firefox now allows you to preview the link and get key points without ever going to the link[1] > [1] https://imgur.com/a/3E17Dts This is generated on device with llama.cpp compiled to webassembly (aka wllama) and running SmolLM2-360M. [1] How is this different from the user clicking on the link? In the end, your local firefox will fetch the link in order to summarize it, the same way you would have followe…
I actually didnt know it was using a local model and that it fetches it locally.
Re: We can't have nice things because of AI scrapers
#136Re: We can't have nice things because of AI scrapers
#137Earlier quoted context omitted.
There is no standard, well-known way for a website to advertise, "hey, here's a cached data dump for bulk download, please use that instead of bulk scraping". If they were, I'd expect the major AI companies and other users[0] to use that method for gathering training data[1]. They have compelling reasons to: it's cheaper for them, and cultivates goodwill instead of burning it. This also means that right now, it could…
You do realize that these AI scrapers are most likely written by people who have no idea what they're doing right? Or they just don't care? If they were, pretty much none of the problems these things have caused would exist. Even if we did standardize such a thing, I doubt they would follow it. After all, they think they and everyone else has infinite resources so they can just hammer websites forever.
Re: We can't have nice things because of AI scrapers
#138My web host suspended my website account last week due to a sudden large volume of requests to it - effectively punishing me for being scraped by bots.
I've had to move to a new host to get back up, but what hope does the little guy have? it's like GPU and ram prices, it doesn't matter if I pay 10x 100x or 1000x more than I did, the AI companies have infinite resources, and they don't care what damage they do in the rush to become the no 1 in the industry
The cynic in me would say it's intentional, destroy all the free sites so you have to get your info from their ai models, price home users out of high end hardware so they have to lease the functions from big companies
Re: We can't have nice things because of AI scrapers
#139Earlier quoted context omitted.
You can implement this yourself, who is stopping you?
Citation needed
They even mention iocaine. I know, inconceivable!: https://iocaine.madhouse-project.org/
There's also tons of HN posts on the topic with varying solutions:
https://news.ycombinator.com/item?id=45935729
https://news.ycombinator.com/item?id=45711094
Re: We can't have nice things because of AI scrapers
#140Earlier quoted context omitted.
Can't seem to access this. It flashes some text briefly then gives me an 418 TEAPOT response. I wonder if it's because I'm on Linux? EDIT: Begrudgingly checked Chrome, and it loads. I guess it doesn't like Firefox?
Doesn't work on my firefox either. Friendly fire, I suppose.