Ask HN: Does ChatGPT respect Robots.txt? • Hacker News Board

Re: Ask HN: Does ChatGPT respect Robots.txt?

#11

Post by wmedrano » Sun, Mar 05, 2023, 11:16 PM UTC

iKevinShah wrote: post #2

To give more context - I asked further: me: If you did not crawl - where did you learn about the data? C: As an AI language model, my learning is based on a pre-existing dataset of text. This dataset is created by training me on a large corpus of text, which can include anything from books and articles to websites and social media posts. This training data is processed and used to develop a model that can understand…

It seems like you are trying to use ChatGPT as a smart entity but it is basically advanced autocomplete. The links it provided are just what is likely to exist, but not what actually does exist

Re: Ask HN: Does ChatGPT respect Robots.txt?

#12

Post by kbelder » Mon, Mar 06, 2023, 2:12 AM UTC

>"I asked ChatGTP..."

"On two occasions I have been asked, 'Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?' I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question."

-Charles Babbage

Re: Ask HN: Does ChatGPT respect Robots.txt?

#13

Post by gtirloni » Mon, Mar 06, 2023, 2:30 AM UTC

amluto wrote: post #9

Why would one expect ChatGPT to know the answer to this question? ChatGPT “knows” what it was trained on. The training data is unlikely to include a definitive answer to your question. And ChatGPT is not currently smart enough to do the kind of analysis that would determine the answer, nor is it likely to be able to do the kind of queries that would be needed to figure it out.

The training data could include internal docs that describe how it ignores or not the robots.txt file.

Re: Ask HN: Does ChatGPT respect Robots.txt?

#14

Post by amluto » Mon, Mar 06, 2023, 3:34 AM UTC

gtirloni wrote: post #13

amluto wrote: post #9

Why would one expect ChatGPT to know the answer to this question? ChatGPT “knows” what it was trained on. The training data is unlikely to include a definitive answer to your question. And ChatGPT is not currently smart enough to do the kind of analysis that would determine the answer, nor is it likely to be able to do the kind of queries that would be needed to figure it out.

The training data could include internal docs that describe how it ignores or not the robots.txt file.

If I were involved at OpenAI, I would not include the internal wiki, Slack archives, Dropbox folders, etc in the training data. While it would be highly entertaining, it would not be a good idea.

Re: Ask HN: Does ChatGPT respect Robots.txt?

#15

Post by JohnFen » Mon, Mar 06, 2023, 5:23 AM UTC

LinuxBender wrote: post #7

I think the best way to answer the question would be to test it out. Have ChatGPT learn something from a URL that is forbidden by robots.txt. FWIW google does not respect robots.txt in the way people think they do. The will still crawl and index a resource but will not publicly display it. Same for archive.org. I've verified that numerous times. Let archive.org index a thing that has always been forbidden by robots.t…

This is why I stopped relying on robots.txt a long time ago. I still use it, but I also have my server check the user agent for crawlers and return a 403 to them.

Re: Ask HN: Does ChatGPT respect Robots.txt?

#16

Post by iKevinShah » Mon, Mar 06, 2023, 6:01 AM UTC

wildpeaks wrote: post #6

Let's not forget that ChatGPT can lie: just because it says something doesn't make it true. It's more realistic to assume that any data a company is able to access will get gobbled up sooner or later because there is no real penalty for ignoring robots.txt or licenses at their scale: even if someone were to notice an infraction and has enough money to sue them for years, they can afford it and brush it off as the cos…

Considering that but about robots.txt is true (and I feel it is true) what can one do. Are there no regulations (implemented or in planning stage) on any of the bodies which decide the standards?

At some point, content owner should be - technically - be having some control to be able to limit / control who accesses their content

Re: Ask HN: Does ChatGPT respect Robots.txt?

#17

Post by iKevinShah » Mon, Mar 06, 2023, 6:04 AM UTC

amluto wrote: post #14

gtirloni wrote: post #13

Earlier quoted context omitted.

The training data could include internal docs that describe how it ignores or not the robots.txt file.

If I were involved at OpenAI, I would not include the internal wiki, Slack archives, Dropbox folders, etc in the training data. While it would be highly entertaining, it would not be a good idea.

I agree on that - that private data (in a best case scenario) should not and would not be included in the training but there would be some parts of internal documents which would be public (lets say public website) - It is expected that chatGPT would know at least those ..