Live data from Hacker News

Go ahead and block AI web crawlers

coryd.dev

21–24 of 24 posts

Re: Go ahead and block AI web crawlers

#21
post #8

It seems quite appropriate that luddites will opt out of the AI training space. Hopefully it will lead to better quality output.

The AI models would not exist without data the so called luddites created.

The best content is not created by luddites. Creative curiousity is generally a necessity for great art.

Re: Go ahead and block AI web crawlers

#22
post #19

Earlier quoted context omitted.

After they ingested everything

We don't actually know what they trained on. If we want to trust them, they've always honored robots.txt, because why wouldn't they? On the other hand, if we want to believe they're as capricious as we want to make them out to be, they've got a copy of LibGen, and every single torrent out there to train it in on, on top of using robots.txt to find what you don't want crawled. robots.txt is great - it tells you where…

Gpt-4 was released to the public on March 14th last year. They told people how to block their agent in August.

Re: Go ahead and block AI web crawlers

#23
post #8

Earlier quoted context omitted.

The AI models would not exist without data the so called luddites created.

The best content is not created by luddites. Creative curiousity is generally a necessity for great art.

I find that people who are willing to not take shortcuts are often the ones making better work, but that's just my opinion.

Re: Go ahead and block AI web crawlers

#24
post #22

Earlier quoted context omitted.

We don't actually know what they trained on. If we want to trust them, they've always honored robots.txt, because why wouldn't they? On the other hand, if we want to believe they're as capricious as we want to make them out to be, they've got a copy of LibGen, and every single torrent out there to train it in on, on top of using robots.txt to find what you don't want crawled. robots.txt is great - it tells you where…

Gpt-4 was released to the public on March 14th last year. They told people how to block their agent in August.

yes but

    User agent: *
Post reply on HN