[dead]
Anyone got a contact at OpenAI. They have a spider problem
61–70 of 400 posts
Re: Anyone got a contact at OpenAI. They have a spider problem
#62Re: Anyone got a contact at OpenAI. They have a spider problem
#63Honeypots like this seem like a super interesting way to poison LLM training.
What exactly is this website? I don’t get it…
A "honeypot" is a system designed to trap unsuspecting entrants. In this case, the website is designed to be found by web crawlers and to then trap them in never-ending linked sites that are all pointless. Other honeypots include things like servers with default passwords designed to be found by hackers so as to find the hackers.
Re: Anyone got a contact at OpenAI. They have a spider problem
#64Isn’t the legality of web scraping still..disputed? There’s been a few projects I’ve wanted to work on involving scraping, but the idea that the entire thing could be shut down with legal threats seems to make some of the ideas infeasible. It’s strange that OpenAI has created a ~$80B company (or whatever it is) using data gathered via scraping and as far as I’m aware there haven’t been any legal threats. Was there so…
Scraping is legal. Always has been, always will be. Mainly because there's some fuzz around the edges of the definition. Is a web browser a scraper? It does a lot of the same things.
IIRC LinkedIn/Microsoft was trying to sue a company based on Computer Fraud and Abuse Act violations, claiming they were accessing information they were not allowed to. Courts ruled that that was bullshit. You can't put up a website and say "you can only look at this with your eyes". Recently-ish, they were found to be in violation of the User Agreement.
So as long as you don't have a user account with the site in question or the site does not have a User Agreement prohibiting scraping, you're golden.
The problem isn't the scraping anyway, it's the reproduction of the work. In that case, it really does matter how you acquired the material and what rights you have with regards use of that material.
Re: Anyone got a contact at OpenAI. They have a spider problem
#65Earlier quoted context omitted.
It's a honeypot. He's telling people openai doesn't respect robots.txt and just scrapes whatever the hell it wants.
It seems to respect it as the majority of the requests are for the robots.txt.
He says 3 million, and 1.8 million are for robots.txt
So 1.2 million non robots.txt requests, when his robots.txt file is configured as follows
# buzz off
User-agent: GPTBot
Disallow: /
Re: Anyone got a contact at OpenAI. They have a spider problem
#66Isn’t the legality of web scraping still..disputed? There’s been a few projects I’ve wanted to work on involving scraping, but the idea that the entire thing could be shut down with legal threats seems to make some of the ideas infeasible. It’s strange that OpenAI has created a ~$80B company (or whatever it is) using data gathered via scraping and as far as I’m aware there haven’t been any legal threats. Was there so…
Web scraping the public Internet is legal, at least in the U.S. hiQ's public scraping of LinkedIn was ruled to be within their rights and not a violation of the CFAA. I imagine that's why LinkedIn has almost everything behind an auth wall now. Scraping auth-walled data is different. When you sign up, you have to check "I agree to the terms," and the terms generally say, "You can't scrape us." So, you can't just make…
hiQ was found to be in violation of the User Agreement in the end.
Basically, in the end, it was essentially a breach of contract.
Re: Anyone got a contact at OpenAI. They have a spider problem
#67Earlier quoted context omitted.
It's a honeypot. He's telling people openai doesn't respect robots.txt and just scrapes whatever the hell it wants.
Except the first thing openai does is read robots.txt. However, robots.txt doesn't cover multiple domains, and every link that's being crawled is to a new domain, which requires a new read of a robots .txt on the new domain.
> Except the fiist thing openai does is read robots.txt.
Then they should see the "Disallow: /" line, which means they shouldn't crawl any links on the page (because even the homepage is disallowed). Which means they wouldn't follow any of the links to other subdomains.
Re: Anyone got a contact at OpenAI. They have a spider problem
#68Earlier quoted context omitted.
It's a honeypot. He's telling people openai doesn't respect robots.txt and just scrapes whatever the hell it wants.
Except the first thing openai does is read robots.txt. However, robots.txt doesn't cover multiple domains, and every link that's being crawled is to a new domain, which requires a new read of a robots .txt on the new domain.
And they do have (the same) robots.txt on every domain, tailored for GPTbot, i.e. https://petra-cody-carlene.web.sp.am/robots.txt
So, GPTBot is not following robots.txt, apparently.
Re: Anyone got a contact at OpenAI. They have a spider problem
#69This reminds me of how GPT-2/3/J came across https://reddit.com/r/counting , wherein redditors repeatedly post incremental numbers to count to infinity. It considered their usernames, like SolidGoldMagikarp, such common strings on the Internet that, during tokenization, it treated them as top-level tokens of their own. https://www.alignmentforum.org/posts/8viQEp8KBg2QSW4Yc/solid... https://www.lesswrong.com/posts/LAx…
During tokenization, the usernames became tokens... but before training the actual model, they removed stuff like that from the training data, so it was never trained on text which contains those tokens. As such, it ended up with tokens which weren't associated with anything; glitch tokens.
Re: Anyone got a contact at OpenAI. They have a spider problem
#70Earlier quoted context omitted.
What exactly is this website? I don’t get it…
A "honeypot" is a system designed to trap unsuspecting entrants. In this case, the website is designed to be found by web crawlers and to then trap them in never-ending linked sites that are all pointless. Other honeypots include things like servers with default passwords designed to be found by hackers so as to find the hackers.
What does trap mean here? I presumed crawlers had multiple (thousands of or more) instances. One being 'trapped' on this web farm won't have any impact