Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI-Shunning robots.txt

github.com

Post reply on HN
Print view View original

31–40 of 92 posts

Re: AI-Shunning robots.txt

#31

Post by Wissenschafter »

vouaobrasil wrote: post #21
cabirum wrote: post #9

The crawlers can simply stop identifying themselves via custom user agent, can't they? Also why are "AI" crawlers are worse than "normal" crawlers? Either way, this is an exercise in futility.

> Either way, this is an exercise in futility. Is it really? Every drop of opposition towards AI in my book is a good thing. This robots.txt thing is a small drop maybe, but over time public hatred for AI can build and it might in fact be taken down. Especially outside the tech bubble, many people are ambivalent towards AI. Yes, in modern society were are taught to value innovation and ignore its downsides, but the m…

Only on hackernews you get the most ironic of takes. Supposedly someone who is educated and technologically literate to a high degree, thinks opposition to AI is a good thing.

Crazy world.

Re: AI-Shunning robots.txt

#32

Post by nerdjon »

Wissenschafter wrote: post #31
vouaobrasil wrote: post #21

Earlier quoted context omitted.

> Either way, this is an exercise in futility. Is it really? Every drop of opposition towards AI in my book is a good thing. This robots.txt thing is a small drop maybe, but over time public hatred for AI can build and it might in fact be taken down. Especially outside the tech bubble, many people are ambivalent towards AI. Yes, in modern society were are taught to value innovation and ignore its downsides, but the m…

Only on hackernews you get the most ironic of takes. Supposedly someone who is educated and technologically literate to a high degree, thinks opposition to AI is a good thing. Crazy world.

> thinks opposition to AI is a good thing.

Because we don't want nearly every job to be automated by AI?

"Crazy world"

Re: AI-Shunning robots.txt

#33

Post by mrkramer »

nerdjon wrote: post #4

I am curious, do we have any evidence that AI is adhering to robots.txt and isn’t ignoring it since they are not technically crawling in the traditional sense? Even if they are right now it would be a quick switch for them to just ignore it.

Internet Archive's crawler is not respecting robots.txt because they want to archive everything not just parts of the Web. But if you are actively breaking robots.txt then your crawler will have a bad reputation and you will have an army of webmasters trying to block your crawler by any means. You can see crawling requests in your sever logs, that's how you know if they are respecting it or not.

Imo, they best solution would be to license your content so crawlers pay a fee for crawling and using your content.

Re: AI-Shunning robots.txt

#34

Post by starbugs »

nerdjon wrote: post #28
cabirum wrote: post #26

Earlier quoted context omitted.

I'm kind of out of the loop in regard why do we need to hate on AI? The bubble will burst given time, like all the other bubbles before. What needed is indifference, not hate.

I think we need to be careful to assume that the AI bubble is really going to burst, there is way too much money being put into it by big players for them to just give up on it. I mean Microsoft is even pushing for a new key on Windows keyboards. We also have to look at this bubble differently. With Crypto there was a general technical level that was required to use it that many people just did not have. I would say…

> Edit: You can really see just how bad this is going to get if you look at Apple. How they are deemed to be "behind" because they don't have generative AI right now and how they are so desperate that they are looking at Google.

> I really don't think this is going to burst and really all that is going to happen is even more consolidation and we will be screwed.

It's always FOMO until it's FUD.

Re: AI-Shunning robots.txt

#35

Post by natch »

We need AIs to know more, not less. If many people block AIs from reading their sites, AIs will just be stuffed with biased information from people pushing agendas.

Re: AI-Shunning robots.txt

#36

Post by CalRobert »

Given how intertwined AI and search engines are it's hard to see how this helps aside from _maybe_ making things easier for Google, Microsoft, etc., unless you also don't want to be indexed by search engines.

Re: AI-Shunning robots.txt

#37

Post by bembo »

nerdjon wrote: post #32
Wissenschafter wrote: post #31

Earlier quoted context omitted.

Only on hackernews you get the most ironic of takes. Supposedly someone who is educated and technologically literate to a high degree, thinks opposition to AI is a good thing. Crazy world.

> thinks opposition to AI is a good thing. Because we don't want nearly every job to be automated by AI? "Crazy world"

Why wouldn't we want jobs to be automated exactly?

Re: AI-Shunning robots.txt

#38

Post by nerdjon »

mrkramer wrote: post #33
nerdjon wrote: post #4

I am curious, do we have any evidence that AI is adhering to robots.txt and isn’t ignoring it since they are not technically crawling in the traditional sense? Even if they are right now it would be a quick switch for them to just ignore it.

Internet Archive's crawler is not respecting robots.txt because they want to archive everything not just parts of the Web. But if you are actively breaking robots.txt then your crawler will have a bad reputation and you will have an army of webmasters trying to block your crawler by any means. You can see crawling requests in your sever logs, that's how you know if they are respecting it or not. Imo, they best soluti…

Well TIL that IA does not respect robots.txt.

Does IA themselves block crawlers? It doesn't look like it according to their robots.txt, even going so far as to say "Please crawl our files."

What would stop an actor from maliciously complying with a robots.txt file by just going to the internet archive instead.

Re: AI-Shunning robots.txt

#40

Post by vouaobrasil »

Wissenschafter wrote: post #31
vouaobrasil wrote: post #21

Earlier quoted context omitted.

> Either way, this is an exercise in futility. Is it really? Every drop of opposition towards AI in my book is a good thing. This robots.txt thing is a small drop maybe, but over time public hatred for AI can build and it might in fact be taken down. Especially outside the tech bubble, many people are ambivalent towards AI. Yes, in modern society were are taught to value innovation and ignore its downsides, but the m…

Only on hackernews you get the most ironic of takes. Supposedly someone who is educated and technologically literate to a high degree, thinks opposition to AI is a good thing. Crazy world.

Actually, I believe it was my education in pure mathematics (PhD), computer programming, and technology, as well as living in three countries and seeing the word that allowed me to truly realize how damaging AI can really be. Believe it or not, I was heavily into technology when I was in my 20s.

But after seeing environmental damage, reading widely in philosophy and sociology, writing about it to clarify it, I came to a different conclusion: that technology is not all it's cracked up to be, especially when it is plugged into a system of global capitalism whose ultimate aim is consumerism.

Just think: one of the biggest companies in the world (Google/Alphabet) has as its primary goal to promote unsustainability. If that doesn't make you think, what will.

And let me ask you this: are you so sure about technology, given that you were raised in a world that praises it like a religion? I think the fanatical religious witch hunters also thought they were right, simply because they were raised in such an environment.

Loving technology is the default position of the rich. Is that just a coincidence?

Post reply on HN