Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI-Shunning robots.txt

github.com

Post reply on HN
Print view View original

81–90 of 92 posts

Re: AI-Shunning robots.txt

#81

Post by glynnormington »

tbeseda wrote: post #79
gavinhking wrote: post #39

Earlier quoted context omitted.

I made this, let me know if you have questions or feedback.

Thanks for the work on this! I automated my site's robots.txt[0] by scraping your site. It would be extra nice if darkvisitor.com exposed a plain text version or JSON representation of the list. [0] https://tbeseda.com/blog/automating-my-robots-txt-to-block-a...

That was one of the points of the new repo. A plain text version of the file is https://raw.githubusercontent.com/ai-robots-txt/ai.robots.tx...

The other point was to make this community maintained rather than rely on one source to provide all the inputs.

Re: AI-Shunning robots.txt

#83

Post by tbeseda »

glynnormington wrote: post #81
tbeseda wrote: post #79

Earlier quoted context omitted.

Thanks for the work on this! I automated my site's robots.txt[0] by scraping your site. It would be extra nice if darkvisitor.com exposed a plain text version or JSON representation of the list. [0] https://tbeseda.com/blog/automating-my-robots-txt-to-block-a...

That was one of the points of the new repo. A plain text version of the file is https://raw.githubusercontent.com/ai-robots-txt/ai.robots.tx... The other point was to make this community maintained rather than rely on one source to provide all the inputs.

Definitely! And I'll likely use that raw URL in my tooling going forward. So thanks for starting the repo.

I'm thinking it also helps to bring up a feature request on the source material so we can all limit the drift. ie. if darkvisitors.com had a sort of plain text API, your repo could check for new entries via GH Actions and create issues or even PRs.

Re: AI-Shunning robots.txt

#84

Post by starbugs »

natch wrote: post #69
nerdjon wrote: post #51

Earlier quoted context omitted.

I am saying the opposite, if they have less data the value of AI's will plummet and hopefully their use will plummet. That is a good thing.

Market dynamics. Since use of better AIs confers advantages, they will be improved. No set of players will be able to stop this because the incentives are so strong for others to continue using and developing them. The best we can hope for is AIs that are not misled. Sometimes you have to work with the tide, because fighting it is futile and even self defeating.

> Sometimes you have to work with the tide, because fighting it is futile and even self defeating.

Said the fish before approaching the waterfall.

While I agree that the incentive structure is set up in a strong way for AI to be further improved and rolled out, what's the endgame here? Who can build the most powerful centralized AI so that nearly everyone else is out of a job? And who is that going to benefit?

I just don't get it.

Have we all decided to "just play the game" and ignore how dumb it is?

Re: AI-Shunning robots.txt

#85

Post by ler_ »

Wissenschafter wrote: post #80
quickslowdown wrote: post #78

Earlier quoted context omitted.

Crazy world when any random person on the Internet just assumes everyone else thinks the way they do. Put me solidly in the camp of educated & "technologically literate to a high degree" and also in opposition of AI.

The two are inherently incompatible to me. I don't understand how you can hold those positions. To me, being against AI is pretty much 'evil'. You support humans existing in a barbaric existence like animals, suffering. I support laws to criminalize people hampering it like they are doing in Japan. I literally will fight against people like you.

Well, with AI people will still likely be suffering, except those at the top of the pyramid will be doing even better. Not so great. The AI debate is not about technology per se, but more so about seeing how gains in technology / productivity only go to the top, and extrapolating how bad that will be when greed meets the hyper-productivity of AI.

Re: AI-Shunning robots.txt

#87

Post by nunez »

Wissenschafter wrote: post #76
nunez wrote: post #55

Earlier quoted context omitted.

Because every job that gets automated creates massive unemployment for those who were skilled in it what do you think will happen to us devs if AI gets good enough to do our jobs? Do you think our companies will keep us around because we're just so darn smart? What do you think is _already happening_ to in-house artists, content/technical writers, marketing analysts, and other jobs that are directly impacted by LLMs…

I'm not a selfish asshole, just a normal asshole. Yeah, I'm going to be affected by my job also being automated, everyone will. It's not a field specific problem. It's a societal paradigm change.

This doesn't bother you? If not, how come?

Re: AI-Shunning robots.txt

#88

Post by omoikane »

nerdjon wrote: post #4

I am curious, do we have any evidence that AI is adhering to robots.txt and isn’t ignoring it since they are not technically crawling in the traditional sense? Even if they are right now it would be a quick switch for them to just ignore it.

I have examples in my logs of GPTBot fetching only /robots.txt, and nothing from the same /24 block fetched anything else after that, so it seems at least that bot respects robots.txt.

Maybe your question is "how do we know if whatever system GPTBot feeds downstream didn't just get your content via something else that crawl your site?" I am not sure we have anything to defend against those, other than signalling via robots.txt to say that our content is not intended for AI use.

Re: AI-Shunning robots.txt

#89

Post by quickslowdown »

Wissenschafter wrote: post #80
quickslowdown wrote: post #78

Earlier quoted context omitted.

Crazy world when any random person on the Internet just assumes everyone else thinks the way they do. Put me solidly in the camp of educated & "technologically literate to a high degree" and also in opposition of AI.

The two are inherently incompatible to me. I don't understand how you can hold those positions. To me, being against AI is pretty much 'evil'. You support humans existing in a barbaric existence like animals, suffering. I support laws to criminalize people hampering it like they are doing in Japan. I literally will fight against people like you.

I'm not going to fight you lol, and your hyperbole means nothing to me. You strut in here making assertions and assume everyone just feels the same and is going to go along.

Your mentality is irresponsible and exactly the problem I have with AI.

Re: AI-Shunning robots.txt

#90

Post by natch »

starbugs wrote: post #84
natch wrote: post #69

Earlier quoted context omitted.

Market dynamics. Since use of better AIs confers advantages, they will be improved. No set of players will be able to stop this because the incentives are so strong for others to continue using and developing them. The best we can hope for is AIs that are not misled. Sometimes you have to work with the tide, because fighting it is futile and even self defeating.

> Sometimes you have to work with the tide, because fighting it is futile and even self defeating. Said the fish before approaching the waterfall. While I agree that the incentive structure is set up in a strong way for AI to be further improved and rolled out, what's the endgame here? Who can build the most powerful centralized AI so that nearly everyone else is out of a job? And who is that going to benefit? I just…

The endgame is the singularity, which, by definition, offers no clear view of the future.

I don’t see any effective way to fight it other than joining it. Any well intentioned steps can be easily and even unintentionally subverted by other players who have different perspectives on the ethical landscape.

It’s not that we have decided. Others will decide for us regardless of our decision.

Post reply on HN