Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sites scramble to block ChatGPT web crawler after instructions emerge

arstechnica.com

Post reply on HN
Print view View original

21–30 of 38 posts

Re: Sites scramble to block ChatGPT web crawler after instructions emerge

#21

Post by ajdude »

TheCaptain4815 wrote: post #20
vouaobrasil wrote: post #16

Earlier quoted context omitted.

If you (or others) come up with a way to build a system to poison AI/LLM/other models to make them useless, count me in to help.

I’d imagine this is best possible via illegal methods such as mass hacking websites and inserting the appropriate poison

Years ago I came across an email crawler trap, where if the bot was unfortunate enough to come across it, it would generate (from the e-mail harvesting bot's point of view) an endless and nested tree of pages with randomly generated garbage emails. It was just a bit of PHP but I wouldn't be surprised if you couldn't hear something that the LLM thinks are comments but It's just randomly generated garbage.

Re: Sites scramble to block ChatGPT web crawler after instructions emerge

#22

Post by aaron695 »

taftster wrote: post #17
vouaobrasil wrote: post #14

From the article: > For example, blocking content from future AI models could decrease a site's or a brand's cultural footprint if AI chatbots become a primary user interface in the future. I would rather leave the internet entirely if AI chatbots become a primary user interface.

What's interesting about your statement is that if you knew they were chatbots, then by definition, they wouldn't be AI. e.g. they wouldn't have passed the Turing test. I know what you're saying, and totally agree. Unfortunately the term "AI" is now meaningless.

[dead]

Re: Sites scramble to block ChatGPT web crawler after instructions emerge

#23

Post by rpgwaiter »

rootw0rm wrote: post #6
JohnFen wrote: post #2

> blocking GPTBot will not guarantee that a site's data does not end up training all AI models of the future. Aside from issues of scrapers ignoring robots.txt files, there are other large data sets of scraped websites (such as The Pile) that are not affiliated with OpenAI. This is why I'm not reassured. robots.txt isn't sufficient to stop all webcrawlers, so there every reason to think it isn't sufficient to stop AI…

There's never going to be a perfect solution, it's an arms race. I really doubt (hope?) that large entities are going to straight up emulate end-user browsers though. I would think filtering based on user agent will be the sweet spot for effort and performance. You could do some awful JavaScript monstrosity to detect the tiny fraction of bots who are sneaky, but if they're determined to be sneaky they will succeed at…

Browsers aren't really trusted platforms, the cool scraping is in emulating phones. Whether that be in actually running a virtual phone or sending traffic that emulates it

Really just encourages phones to be even more locked down

Re: Sites scramble to block ChatGPT web crawler after instructions emerge

#24

Post by vouaobrasil »

taftster wrote: post #17
vouaobrasil wrote: post #14

From the article: > For example, blocking content from future AI models could decrease a site's or a brand's cultural footprint if AI chatbots become a primary user interface in the future. I would rather leave the internet entirely if AI chatbots become a primary user interface.

What's interesting about your statement is that if you knew they were chatbots, then by definition, they wouldn't be AI. e.g. they wouldn't have passed the Turing test. I know what you're saying, and totally agree. Unfortunately the term "AI" is now meaningless.

True enough, but one doesn't need to know whether they are Chatbots. In fact, one doesn't even have to have ever heard of AI/ChatGPT to know that something fundamnetally inhuman and strange were going on. If a person woke up from a coma after Chatbots had taken over the internet, they would immediately notice something very strange and machine-like about the whole experience.

Re: Sites scramble to block ChatGPT web crawler after instructions emerge

#25

Post by CableNinja »

JohnFen wrote: post #2

> blocking GPTBot will not guarantee that a site's data does not end up training all AI models of the future. Aside from issues of scrapers ignoring robots.txt files, there are other large data sets of scraped websites (such as The Pile) that are not affiliated with OpenAI. This is why I'm not reassured. robots.txt isn't sufficient to stop all webcrawlers, so there every reason to think it isn't sufficient to stop AI…

I chose to use an nginx entry, because i also dont trust them to follow robots.txt. Throwing a 410 Gone should keep them from coming back too, theoretically, assuming they actually eject when receiving it, like it should.

`if ($http_user_agent ~* ".*?(GPTBot|AI).*?") { return 410; }`

Its not perfect, but it should filter them indefinitely, will probably have to add some more terms in there over time.

Re: Sites scramble to block ChatGPT web crawler after instructions emerge

#26

Post by whelp_24 »

taftster wrote: post #17
vouaobrasil wrote: post #14

From the article: > For example, blocking content from future AI models could decrease a site's or a brand's cultural footprint if AI chatbots become a primary user interface in the future. I would rather leave the internet entirely if AI chatbots become a primary user interface.

What's interesting about your statement is that if you knew they were chatbots, then by definition, they wouldn't be AI. e.g. they wouldn't have passed the Turing test. I know what you're saying, and totally agree. Unfortunately the term "AI" is now meaningless.

That's not what AI means. Also chatgpt can beat the Turing test often enough.

Re: Sites scramble to block ChatGPT web crawler after instructions emerge

#27

Post by CableNinja »

For anyone reading this, you can skip the robots.txt, as others have pointed out, who knows if they will actually listen to it.

Instead, use a redirect or return a response code by doing a user agent check in your server config. I posted elsewhere in this thread on the way i did it with nginx

Re: Sites scramble to block ChatGPT web crawler after instructions emerge

#28

Post by salawat »

CableNinja wrote: post #27

For anyone reading this, you can skip the robots.txt, as others have pointed out, who knows if they will actually listen to it. Instead, use a redirect or return a response code by doing a user agent check in your server config. I posted elsewhere in this thread on the way i did it with nginx

...You realize user agent is operator set, right?

If they won't reapect robots.txt, they aren't interested in your consent.

Re: Sites scramble to block ChatGPT web crawler after instructions emerge

#30

Post by danielbln »

taftster wrote: post #17
vouaobrasil wrote: post #14

From the article: > For example, blocking content from future AI models could decrease a site's or a brand's cultural footprint if AI chatbots become a primary user interface in the future. I would rather leave the internet entirely if AI chatbots become a primary user interface.

What's interesting about your statement is that if you knew they were chatbots, then by definition, they wouldn't be AI. e.g. they wouldn't have passed the Turing test. I know what you're saying, and totally agree. Unfortunately the term "AI" is now meaningless.

Complaining about overloading the word "AI" is like complaining about the term "Cloud". Might as well move past it.
Post reply on HN