Frankly, I didn’t get the purpose of the website at first either. I guess I have an arachnid intellect.
You had the sense not to spend the rest of your days crawling it though.
Anyone got a contact at OpenAI. They have a spider problem
191–200 of 400 posts
Re: Anyone got a contact at OpenAI. They have a spider problem
#192Eventually, OpenAI (and friends) are going to be training their models on almost exclusively AI generated content, which is more often than not slightly incorrect when it comes to Q&A, and the quality of AI responses trained on that content will quickly deteriorate. Right now, most internet content is written by humans. But in 5 years? Not so much. I think this is one of the big problems that the AI space needs to so…
As long as humans continue to filter out the bad content generated by AI, it should be fine.
Re: Anyone got a contact at OpenAI. They have a spider problem
#193Earlier quoted context omitted.
That’s a good observation. If LLMs had taken off 15 years ago, maybe they would answer every question with “this has already been asked before. Please use the search function”
Nowadays they are instead learning to say "please join our Discord for support"!
Much like one of the first phrases spoken by babies today is "like and subscribe".
Re: Anyone got a contact at OpenAI. They have a spider problem
#194Re: Anyone got a contact at OpenAI. They have a spider problem
#195Earlier quoted context omitted.
humans don't read/respect robots.txt, so in order to pass the Turing test, ai's need to mimic human behavior.
This must be why self-driving cars always ignore the speed limit. ;)
More directly, e.g. Tesla boasts of training their FSD on data captured from their customer's unassisted driving. So it's hardly surprising that it imitates a lot of humans' bad habits, e.g. rolling past stop lines.
Re: Anyone got a contact at OpenAI. They have a spider problem
#196Earlier quoted context omitted.
I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs. I mean if someone posts a question on an internet forum that I don't know the answer to, I'm certainly not going to post "I don't know" since that wouldn't be useful. In fact, in general, in any non one-on-one conversation the answer "I don't know" is not useful because if you don't know in a group, your silence…
I've wondered if one could train a LLM on a closed set of curated knowledge. Then include training data that models the behaviour of not knowing. To the point that it could generalize to being able to represent its own not knowing. Because expecting a behaviour, like knowing you don't know, that isn't represented in the training set is silly. Kids make stuff up at first, then we correct them - so they have a way to l…
> I've wondered if one could train a LLM on a closed set of curated knowledge. Then include training data that models the behaviour of not knowing. To the point that it could generalize to being able to represent its own not knowing.
The problem is that curating data is slow and expensive and downloading the entire web is fast and cheap.
See also https://en.wikipedia.org/wiki/Cyc
Re: Anyone got a contact at OpenAI. They have a spider problem
#197Earlier quoted context omitted.
> Facebook alone probably has more data than the entire dataset GPT4 was trained on and it’s all behind closed doors. Meta is happily training their own models with this data, so it isn't going to waste.
Not Llama, they’ve been really clear about that. Especially with DMA cross-joining provisions and various privacy requirements it’s really hard for them, same for Google. However, Microsoft has been flying under the radar. If they gave all Hotmail and O365 data to OpenAI I’d not be surprised in the slightest.
The company that made a honeypot VPN to access competitor's traffic? They are definitively keeping their hands off their internal data, yes.
Re: Anyone got a contact at OpenAI. They have a spider problem
#198Earlier quoted context omitted.
Got a source for that?
Want a real conspiracy? What do you think the NSA is storing in that datacenter in Utah? Power point presentations? All that data is going to be trained into large models. Every phone call you ever had and every email you ever wrote. They are likely pumping enormous money into it as we speak, probably with the help of OpenAI, Microsoft and friends.
Though it seems like something that could exist, who is doing the technical work/programming? It seems impossible to be in the industry and not have associates and colleagues either from or going to an operation like that. This is what I've always pondered about when it comes to any idea like this. The number of engineers at the pointy end of the tech spear is pretty small.
Re: Anyone got a contact at OpenAI. They have a spider problem
#199This reminds me of how GPT-2/3/J came across https://reddit.com/r/counting , wherein redditors repeatedly post incremental numbers to count to infinity. It considered their usernames, like SolidGoldMagikarp, such common strings on the Internet that, during tokenization, it treated them as top-level tokens of their own. https://www.alignmentforum.org/posts/8viQEp8KBg2QSW4Yc/solid... https://www.lesswrong.com/posts/LAx…
I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs. I mean if someone posts a question on an internet forum that I don't know the answer to, I'm certainly not going to post "I don't know" since that wouldn't be useful. In fact, in general, in any non one-on-one conversation the answer "I don't know" is not useful because if you don't know in a group, your silence…
"I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs."
Probably true, but if you have quality, organized data, you will just want to search the data itself.
Re: Anyone got a contact at OpenAI. They have a spider problem
#200Earlier quoted context omitted.
This would be considered a Slow Loris attack, and I'm actually curious how scrapers would handle it. I'm sure the big players like Google would deal with it gracefully.
Here you go (1 req/min, 10 bytes/sec), please report results :) http { limit_req_zone $binary_remote_addr zone=ten_bytes_per_second:10m rate=1r/m; server { location / { if ($http_user_agent = "mimo") { limit_req zone=ten_bytes_per_second burst=5; limit_rate 10; } } } }
Scrapers of the future won't be ifElse logic, they will be LLM agents themselves. The slow loris robots.txt has to provide an interface to it's own LLM, which engages the scraper LLM in conversation, aiming to extend it as long as possible.
"OK I will tell you whether or not I can be scraped. BUT FIRST, listen to this offer. I can give you TWO SCRAPES instead of one, if you can solve this riddle."