Honestly, that seems like an excellent opportunity to feed garbage into OpenAI's training process.
Anyone got a contact at OpenAI. They have a spider problem
81–90 of 400 posts
Re: Anyone got a contact at OpenAI. They have a spider problem
#82Earlier quoted context omitted.
Except the first thing openai does is read robots.txt. However, robots.txt doesn't cover multiple domains, and every link that's being crawled is to a new domain, which requires a new read of a robots .txt on the new domain.
Did we just figure out a DoS attack for AGI training? How large can a robots.txt file be?
No, because there’s no legal weight behind robots.txt.
The second someone weaponizes robots.txt all the scrapers will just start ignoring it.
Re: Anyone got a contact at OpenAI. They have a spider problem
#83Eventually, OpenAI (and friends) are going to be training their models on almost exclusively AI generated content, which is more often than not slightly incorrect when it comes to Q&A, and the quality of AI responses trained on that content will quickly deteriorate. Right now, most internet content is written by humans. But in 5 years? Not so much. I think this is one of the big problems that the AI space needs to so…
It's true that there will no longer be any virgin forest to scrape but it's also true that content humans want will still be most popular and promoted and curated and edited etc etc. Even if it's impossible to train on organic content it'll still be possible to get good content
Re: Anyone got a contact at OpenAI. They have a spider problem
#84The website is 12 years old and explained here -
https://circleid.com/posts/20120713_silly_bing
John Levine is a known name in IT. Probably best know on HN as the author of "UNIX For Dummies"
Re: Anyone got a contact at OpenAI. They have a spider problem
#85With all the news about scraping legality you'd think a multi billion dollar AI company would try to obfuscate their attempts.
Re: Anyone got a contact at OpenAI. They have a spider problem
#86This reminds me of how GPT-2/3/J came across https://reddit.com/r/counting , wherein redditors repeatedly post incremental numbers to count to infinity. It considered their usernames, like SolidGoldMagikarp, such common strings on the Internet that, during tokenization, it treated them as top-level tokens of their own. https://www.alignmentforum.org/posts/8viQEp8KBg2QSW4Yc/solid... https://www.lesswrong.com/posts/LAx…
I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs. I mean if someone posts a question on an internet forum that I don't know the answer to, I'm certainly not going to post "I don't know" since that wouldn't be useful.
In fact, in general, in any non one-on-one conversation the answer "I don't know" is not useful because if you don't know in a group, your silence indicates that.
Re: Anyone got a contact at OpenAI. They have a spider problem
#87Re: Anyone got a contact at OpenAI. They have a spider problem
#88Earlier quoted context omitted.
A "honeypot" is a system designed to trap unsuspecting entrants. In this case, the website is designed to be found by web crawlers and to then trap them in never-ending linked sites that are all pointless. Other honeypots include things like servers with default passwords designed to be found by hackers so as to find the hackers.
What does trap mean here? I presumed crawlers had multiple (thousands of or more) instances. One being 'trapped' on this web farm won't have any impact
In this case there are >6bn pages with roughly zero value each. That could eat a substantial amount of time. It's unlikely to entirely trap a crawler, but a dumb crawler (as is implied here) will start crawling more and more pages, becoming very apparent to the operator of this honeypot (and therefore identifying new crawlers), and may take up more and more share of the crawl set.
Re: Anyone got a contact at OpenAI. They have a spider problem
#89My assumption is that OpenAI reads the robots.txt, but indexes anyway; they just make a note of what content they weren't supposed to index.
Re: Anyone got a contact at OpenAI. They have a spider problem
#90Isn’t the legality of web scraping still..disputed? There’s been a few projects I’ve wanted to work on involving scraping, but the idea that the entire thing could be shut down with legal threats seems to make some of the ideas infeasible. It’s strange that OpenAI has created a ~$80B company (or whatever it is) using data gathered via scraping and as far as I’m aware there haven’t been any legal threats. Was there so…
There’s currently only one situation where scraping is almost definitely “not legal”: If the information you’re scraping requires a login, and if in order to get a login you have to agree to a terms of service, and that terms of service forbids you from scraping — then you could have a bad day in civil court if the website you’re scraping decides to sue you. If the data is publicly accessible without a login then scr…
Also worth noting there's a long history of companies with deep pockets getting away with murder (sometimes literally) because litigation in a system that costs money to engage with inherently favors the wealthier party.
Also OpenAI's entire business model is relying on generous interpretations of various IP laws, so I suspect they already have a mature legal division to handle these sorts of potential issues.