Anyone got a contact at OpenAI. They have a spider problem

Post by matt_heimer » Thu, Apr 11, 2024, 3:30 PM UTC

apocalyptic0n3 wrote: post #28

unsupp0rted wrote: post #7

Earlier quoted context omitted.

> Before someone tells me to fix my robots.txt, this is a content farm so rather than being one web site with 6,859,000,000 pages, it is 6,859,000,000 web sites each with one page.

The reason that bit is relevant is that robots.txt is only applicable to the current domain. Because each "page" is a different subdomain, the crawler needs to fetch the robots.txt for every single page request. What the poster was suggesting is blocking them at a higher level - e.g. a user-agent block in an .htaccess or an IP block in iptables or similar. That would be a one-stop fix. It would also defeat the purpos…

  1. GPTBot treats a disallow as a noindex but still requests the page itself. Note that Google doesn't treat a disallow as a noindex. They will still show your page in search results if they discover the link from other pages but they show it with a "No information is available for this page." disclaimer.
  2. The site didn't have a GPTBot disallow until they noticed the traffic spike and they bot has already discovered a couple million links that need to be crawled.
  3. There is some other page out there on the internet that GPTBot discovered that links to millions of these subdomains. This seems possible and the subdomains really don't have any way to prevent a bot from requesting millions of robots.txt files. The only prevention here is to firewall the bot's IP range or work with the bot owners to implement better subdomain handling.

Post by SirMaster » Thu, Apr 11, 2024, 3:34 PM UTC

abracadaniel wrote: post #91

aidenn0 wrote: post #86

Earlier quoted context omitted.

I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs. I mean if someone posts a question on an internet forum that I don't know the answer to, I'm certainly not going to post "I don't know" since that wouldn't be useful. In fact, in general, in any non one-on-one conversation the answer "I don't know" is not useful because if you don't know in a group, your silence…

That’s a good observation. If LLMs had taken off 15 years ago, maybe they would answer every question with “this has already been asked before. Please use the search function”

Post by zelphirkalt » Thu, Apr 11, 2024, 3:36 PM UTC

mort96 wrote: post #69

btown wrote: post #30

This reminds me of how GPT-2/3/J came across https://reddit.com/r/counting , wherein redditors repeatedly post incremental numbers to count to infinity. It considered their usernames, like SolidGoldMagikarp, such common strings on the Internet that, during tokenization, it treated them as top-level tokens of their own. https://www.alignmentforum.org/posts/8viQEp8KBg2QSW4Yc/solid... https://www.lesswrong.com/posts/LAx…

During tokenization, the usernames became tokens... but before training the actual model, they removed stuff like that from the training data, so it was never trained on text which contains those tokens. As such, it ended up with tokens which weren't associated with anything; glitch tokens.

Post by [deleted] » Thu, Apr 11, 2024, 3:37 PM UTC

cwillu wrote: post #55

Takennickname wrote: post #50

Earlier quoted context omitted.

It's a honeypot. He's telling people openai doesn't respect robots.txt and just scrapes whatever the hell it wants.

Except the first thing openai does is read robots.txt. However, robots.txt doesn't cover multiple domains, and every link that's being crawled is to a new domain, which requires a new read of a robots .txt on the new domain.

Post by khalladay » Thu, Apr 11, 2024, 3:43 PM UTC

sarah_eu wrote: post #102

pants2 wrote: post #99

Earlier quoted context omitted.

OpenAI will just litter microphones around public spaces to record conversations and train on them.

Has been happening for at least 10 years.