Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI-Shunning robots.txt

github.com

Post reply on HN
Print view View original

21–30 of 92 posts

Re: AI-Shunning robots.txt

#21

Post by vouaobrasil »

cabirum wrote: post #9

The crawlers can simply stop identifying themselves via custom user agent, can't they? Also why are "AI" crawlers are worse than "normal" crawlers? Either way, this is an exercise in futility.

> Either way, this is an exercise in futility.

Is it really? Every drop of opposition towards AI in my book is a good thing. This robots.txt thing is a small drop maybe, but over time public hatred for AI can build and it might in fact be taken down. Especially outside the tech bubble, many people are ambivalent towards AI.

Yes, in modern society were are taught to value innovation and ignore its downsides, but the more vocal opponents are against it, the more those downsides will become apparent. Hopefully, it will bring the ruin of all AI companies and research.

Re: AI-Shunning robots.txt

#22

Post by gtirloni »

rideontime wrote: post #15
rocky_raccoon wrote: post #14

Not that I'm arguing for or against preventing access from AI crawlers, but wouldn't it make more sense to block them at a higher level, e.g. the webserver, and not even give them the choice to obey/disobey robots.txt?

How would you propose doing so?

Web servers can check the user-agent and block the request.

E.g. nginx $http_user_agent

Re: AI-Shunning robots.txt

#23

Post by vouaobrasil »

belter wrote: post #3

Or redirect them to poisoned material?

That is a good idea. Maybe redirect them to massive datasets to cause the company mass embarrassment. There are already some image-modifying programs that generate poison images, and the bots could be redirected to such images...

Re: AI-Shunning robots.txt

#24

Post by vouaobrasil »

rideontime wrote: post #17
vouaobrasil wrote: post #11

Earlier quoted context omitted.

> How about AI from non-companies? Or genuinely non-profit or open projects? AI from any project will allow AI to be used commercially, and thus I oppose it. Moreover, I oppose AI on various other princincples even independent of this: it further isolates people and can be used to develop other technologies that are too powerful for us to handle. In short, I believe human beings en mass are too stupid to use AI. > Al…

Likewise, I've unsubscribed from multiple paid Patreons and Substacks as soon as they started using AI to generate content. I'd rather see an amateur MSPaint scribble than some dall-e monstrosity at the head of a newsletter.

Exactly! Even if a person isn't that great at drawing or painting, it's so much more interesting to view their attempts because they reflect something about the person. The AI-generated fluff might look nice (even I'll admit some look interesting), but it's devoid of human soul. And there's really not much point in art if it has no communicative value that originates from living beings.

Re: AI-Shunning robots.txt

#25

Post by nerdjon »

vouaobrasil wrote: post #11
andybak wrote: post #7

Earlier quoted context omitted.

How about AI from non-companies? Or genuinely non-profit or open projects? Also - out of curiosity - do you use any AI yourself?

> How about AI from non-companies? Or genuinely non-profit or open projects? AI from any project will allow AI to be used commercially, and thus I oppose it. Moreover, I oppose AI on various other princincples even independent of this: it further isolates people and can be used to develop other technologies that are too powerful for us to handle. In short, I believe human beings en mass are too stupid to use AI. > Al…

I think this is an interesting situation where “AI” has become just a general term that it has lost so much meaning thanks to things like chatgpt.

Video game AI is obviously in a different league than ChatGPT but uses the same label.

Some AI is machine learning and some isn’t.

I agree with a lot of what you are saying, but I think there are valid use cases of AI (before chatgpt) that is actually a benefit.

You don’t use a smart phone, but auto correct is a genuinely great addition. It doesn’t remove anything the human does and improves the usability.

On its own autocorrect isn’t going to write a story. Even the suggestions that have been added in recent years are more for human usability than anything else.

Handwriting reading models, fall detection models, etc.

I do think we need to separate generative AI that replaces humans from traditional AI that offers assistance.

I know someone is going to argue, well chatgpt is augmenting me by checking my code, emails, etc. and that may be true right now but we are kidding ourselves if that will be the situation long term.

Re: AI-Shunning robots.txt

#26

Post by cabirum »

vouaobrasil wrote: post #21
cabirum wrote: post #9

The crawlers can simply stop identifying themselves via custom user agent, can't they? Also why are "AI" crawlers are worse than "normal" crawlers? Either way, this is an exercise in futility.

> Either way, this is an exercise in futility. Is it really? Every drop of opposition towards AI in my book is a good thing. This robots.txt thing is a small drop maybe, but over time public hatred for AI can build and it might in fact be taken down. Especially outside the tech bubble, many people are ambivalent towards AI. Yes, in modern society were are taught to value innovation and ignore its downsides, but the m…

I'm kind of out of the loop in regard why do we need to hate on AI? The bubble will burst given time, like all the other bubbles before.

What needed is indifference, not hate.

Re: AI-Shunning robots.txt

#27

Post by starbugs »

cabirum wrote: post #26
vouaobrasil wrote: post #21

Earlier quoted context omitted.

> Either way, this is an exercise in futility. Is it really? Every drop of opposition towards AI in my book is a good thing. This robots.txt thing is a small drop maybe, but over time public hatred for AI can build and it might in fact be taken down. Especially outside the tech bubble, many people are ambivalent towards AI. Yes, in modern society were are taught to value innovation and ignore its downsides, but the m…

I'm kind of out of the loop in regard why do we need to hate on AI? The bubble will burst given time, like all the other bubbles before. What needed is indifference, not hate.

> What needed is indifference, not hate.

Agree that hate is not the solution. But saying "no" decidedly and loudly is absolutely something that people need to relearn and I am happy to see that some do now.

Just being indifferent is not sufficient.

Finding and taking measures to protect yourself and others is a positive way to approach this IMO.

Re: AI-Shunning robots.txt

#28

Post by nerdjon »

cabirum wrote: post #26
vouaobrasil wrote: post #21

Earlier quoted context omitted.

> Either way, this is an exercise in futility. Is it really? Every drop of opposition towards AI in my book is a good thing. This robots.txt thing is a small drop maybe, but over time public hatred for AI can build and it might in fact be taken down. Especially outside the tech bubble, many people are ambivalent towards AI. Yes, in modern society were are taught to value innovation and ignore its downsides, but the m…

I'm kind of out of the loop in regard why do we need to hate on AI? The bubble will burst given time, like all the other bubbles before. What needed is indifference, not hate.

I think we need to be careful to assume that the AI bubble is really going to burst, there is way too much money being put into it by big players for them to just give up on it. I mean Microsoft is even pushing for a new key on Windows keyboards.

We also have to look at this bubble differently. With Crypto there was a general technical level that was required to use it that many people just did not have. I would say the same is true for most bubbles we have seen.

AI is generally available for anyone that has enough understanding on how to open a browser to use. Too many non technical people have just accepted it without understanding the risks.

Edit: You can really see just how bad this is going to get if you look at Apple. How they are deemed to be "behind" because they don't have generative AI right now and how they are so desperate that they are looking at Google.

I really don't think this is going to burst and really all that is going to happen is even more consolidation and we will be screwed.

Re: AI-Shunning robots.txt

#29

Post by vouaobrasil »

cabirum wrote: post #26
vouaobrasil wrote: post #21

Earlier quoted context omitted.

> Either way, this is an exercise in futility. Is it really? Every drop of opposition towards AI in my book is a good thing. This robots.txt thing is a small drop maybe, but over time public hatred for AI can build and it might in fact be taken down. Especially outside the tech bubble, many people are ambivalent towards AI. Yes, in modern society were are taught to value innovation and ignore its downsides, but the m…

I'm kind of out of the loop in regard why do we need to hate on AI? The bubble will burst given time, like all the other bubbles before. What needed is indifference, not hate.

I disagree. If something is truly damaging to society, we need to opposite it actively and crush it.

Indifference is what leads people to ignore climate change. Indifference is what leads people to allow corporations to destroy communities. Indifference is what allows global capitalism to keep going.

I am not concerned with an AI hype bubble bursting. AI already has enough power to strengthen the industrial society outside the hype. Perhaps hatred is not quite the right word. Perhaps love is a better one: a love for smaller communities and more sustainability. Wasn't it Che Guevara who said, "the true revolutionary is guided by strong feelings of love."?

Regardless, indifference is not he answer: it is passionate opposition with a zero tolerance policy towards AI, AI researchers, and AI companies.

Re: AI-Shunning robots.txt

#30

Post by cabirum »

karaterobot wrote: post #18
cabirum wrote: post #9

The crawlers can simply stop identifying themselves via custom user agent, can't they? Also why are "AI" crawlers are worse than "normal" crawlers? Either way, this is an exercise in futility.

> Also why are "AI" crawlers are worse than "normal" crawlers? A search engine will index your content to bring people to it through search. An AI crawler will take your content to recapitulate it and sell it to others. Obviously it's more complicated than this, but this is how one might see it who wishes to use this file. > Either way, this is an exercise in futility. Not necessarily disqualifying. Laws against thef…

Search engines are not the beacons of holiness - they sell ads, they sell data on who searched what, they manipulate results.

Search engines and AI things are typically owned by the same company. AIs are fed with the data collected by a search engine. The only difference is whether AI gets the data in realtime or waits for the search engine to collect another data dump.

Fighting windmills as I see it.

Post reply on HN