Live data from Hacker News

AI-Shunning robots.txt

github.com

41–50 of 92 posts

Re: AI-Shunning robots.txt

#41
post #37
post #32

Earlier quoted context omitted.

> thinks opposition to AI is a good thing. Because we don't want nearly every job to be automated by AI? "Crazy world"

Why wouldn't we want jobs to be automated exactly?

Well at this point in time we live in a capitalist world and people need money to survive?

I assume you like being paid, buying things, food, etc.

Would it be great if we lived in a utopian society that money no longer mattered. Sure! Even with AI I see basically zero chance of that happening in any reasonable amount of time before AI destroys our society.

Re: AI-Shunning robots.txt

#42
post #25

Earlier quoted context omitted.

> How about AI from non-companies? Or genuinely non-profit or open projects? AI from any project will allow AI to be used commercially, and thus I oppose it. Moreover, I oppose AI on various other princincples even independent of this: it further isolates people and can be used to develop other technologies that are too powerful for us to handle. In short, I believe human beings en mass are too stupid to use AI. > Al…

I think this is an interesting situation where “AI” has become just a general term that it has lost so much meaning thanks to things like chatgpt. Video game AI is obviously in a different league than ChatGPT but uses the same label. Some AI is machine learning and some isn’t. I agree with a lot of what you are saying, but I think there are valid use cases of AI (before chatgpt) that is actually a benefit. You don’t…

>I think this is an interesting situation where “AI” has become just a general term that it has lost so much meaning thanks to things like chatgpt.

AI is just a branding buzzword by Big Tech companies....before AI, there were "machine learning" and "deep learning" buzzwords widely used. "Blockchain" and "SaaS" are also some of the most infamous buzzwords.

Re: AI-Shunning robots.txt

#43
post #7

Earlier quoted context omitted.

How about AI from non-companies? Or genuinely non-profit or open projects? Also - out of curiosity - do you use any AI yourself?

> How about AI from non-companies? Or genuinely non-profit or open projects? AI from any project will allow AI to be used commercially, and thus I oppose it. Moreover, I oppose AI on various other princincples even independent of this: it further isolates people and can be used to develop other technologies that are too powerful for us to handle. In short, I believe human beings en mass are too stupid to use AI. > Al…

OK - that's fine then. I might disagree but I can't accuse you of inconsistency.

Re: AI-Shunning robots.txt

#44
post #37
post #32

Earlier quoted context omitted.

> thinks opposition to AI is a good thing. Because we don't want nearly every job to be automated by AI? "Crazy world"

Why wouldn't we want jobs to be automated exactly?

Because if a job is automted, it means that the person whose job is automated now is unemployed. Even if we have UBI, it means that the person DOING the automating will get a disproportionate share of the resources compared with the pittance given to the person who was automated.

Personally, I don't want my job to be automated. I write for a living and if AI takes my job, I won't get paid. I prefer to create value in the world that other people appreciate. I don't WANT to sit in a concrete cage (an apartment) and consume media, with no real purpose in society.

Believe it or not, the majority of people in the world need to feel like they are working for something. Yes, some people will be able to find other causes (mine will be the opposition of AI), but others won't. Of course, that will mean the necessity of drugging people with media (and physical substances...why do you think marijuana is becoming legal in more places?).

The end result is a mode of pure consumption for almost all except the elite who control all the production, and they will decide what happens with the world. Personally, I don't want that: I want land and autonomy to use it to grow food and preserve ecosystems. I want the world to be sustainable, and not just set up for the purpose of furthering technology.

You speak of societal changes on a year-scale. I'm talking about decades and the long-term. This level of automation is bad, and won't do any favours for humanity except the ultra-rich, who will eventually perish like everyone else.

Re: AI-Shunning robots.txt

#45

Earlier quoted context omitted.

> Either way, this is an exercise in futility. Is it really? Every drop of opposition towards AI in my book is a good thing. This robots.txt thing is a small drop maybe, but over time public hatred for AI can build and it might in fact be taken down. Especially outside the tech bubble, many people are ambivalent towards AI. Yes, in modern society were are taught to value innovation and ignore its downsides, but the m…

Only on hackernews you get the most ironic of takes. Supposedly someone who is educated and technologically literate to a high degree, thinks opposition to AI is a good thing. Crazy world.

> Supposedly someone who is educated and technologically literate to a high degree, thinks opposition to AI is a good thing.

Ignoring that your comment is phrased in a hurtful way towards the parent which got you a downvote from me, why do you think these two are connected in any way?

Re: AI-Shunning robots.txt

#46
post #25

Earlier quoted context omitted.

I think this is an interesting situation where “AI” has become just a general term that it has lost so much meaning thanks to things like chatgpt. Video game AI is obviously in a different league than ChatGPT but uses the same label. Some AI is machine learning and some isn’t. I agree with a lot of what you are saying, but I think there are valid use cases of AI (before chatgpt) that is actually a benefit. You don’t…

>I think this is an interesting situation where “AI” has become just a general term that it has lost so much meaning thanks to things like chatgpt. AI is just a branding buzzword by Big Tech companies....before AI, there were "machine learning" and "deep learning" buzzwords widely used. "Blockchain" and "SaaS" are also some of the most infamous buzzwords.

I get what you are saying, but AI isn't a new term. Even when ML was a commonly used term AI was still thrown around.

For as long as I can remember AI has been the term used for video games.

But that is the problem when talking about "AI". Because Video Game AI isn't even Machine learning, but we also call ChatGPT AI.

So I just caution against saying "No AI" when that could get rid of things that are nowhere near what we are currently talking about for AI.

Re: AI-Shunning robots.txt

#47
post #35

We need AIs to know more, not less. If many people block AIs from reading their sites, AIs will just be stuffed with biased information from people pushing agendas.

So the value of them will plummet? That sounds like a win for society.

Re: AI-Shunning robots.txt

#48
post #38

Earlier quoted context omitted.

Internet Archive's crawler is not respecting robots.txt because they want to archive everything not just parts of the Web. But if you are actively breaking robots.txt then your crawler will have a bad reputation and you will have an army of webmasters trying to block your crawler by any means. You can see crawling requests in your sever logs, that's how you know if they are respecting it or not. Imo, they best soluti…

Well TIL that IA does not respect robots.txt. Does IA themselves block crawlers? It doesn't look like it according to their robots.txt, even going so far as to say "Please crawl our files." What would stop an actor from maliciously complying with a robots.txt file by just going to the internet archive instead.

>Well TIL that IA does not respect robots.txt.

At least, that's what they say[0].

>What would stop an actor from maliciously complying with a robots.txt file by just going to the internet archive instead.

Nothing; as far as I understand scraping public web is legal or that's what courts are saying lately. Btw, it's mind boggling to be me that after 30 years of commercial Internet and Web, we still don't have a definite answer is scraping of public websites and public web content legal or illegal.

[0] https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...

Re: AI-Shunning robots.txt

#49
post #47
post #35

We need AIs to know more, not less. If many people block AIs from reading their sites, AIs will just be stuffed with biased information from people pushing agendas.

So the value of them will plummet? That sounds like a win for society.

Why would the value of AIs plummet if they know more?

Or did you mean sites? Information wants to be free.

If AI is trained only on data provided by those with agendas, you won’t want to live in that world.

Re: AI-Shunning robots.txt

#50
post #38

Earlier quoted context omitted.

Well TIL that IA does not respect robots.txt. Does IA themselves block crawlers? It doesn't look like it according to their robots.txt, even going so far as to say "Please crawl our files." What would stop an actor from maliciously complying with a robots.txt file by just going to the internet archive instead.

>Well TIL that IA does not respect robots.txt. At least, that's what they say[0]. >What would stop an actor from maliciously complying with a robots.txt file by just going to the internet archive instead. Nothing; as far as I understand scraping public web is legal or that's what courts are saying lately. Btw, it's mind boggling to be me that after 30 years of commercial Internet and Web, we still don't have a defini…

> Nothing; as far as I understand scraping public web is legal or that's what courts are saying lately. Btw, it's mind boggling to be me that after 30 years of commercial Internet and Web, we still don't have a definite answer is scraping of public websites and public web content legal or illegal.

I was more thinking from a public perception side instead of legal, but legal would be a good question too.

Something like, "Yeah I totally respected your robots.txt file the only reason I have your data is because I crawled IA, see they are the ones you should be mad at not us"

Post reply on HN