I am wondering if amazon fixed the issue or blacklisted *.sp.am
Anyone got a contact at OpenAI. They have a spider problem
151–160 of 400 posts
Re: Anyone got a contact at OpenAI. They have a spider problem
#152Earlier quoted context omitted.
The end state of training on web text has always been an ouroboros - primarily because of adtech incentives to produce low quality content at scale to capture micro pennies. The irony of the whole thing is brutal.
Content you’re allowed and capable of scraping on the Internet is such a small amount of data, not sure why people are acting otherwise. Common crawl alone is only a few hundred TB, I have more content than that on a NAS sitting in my office that I built for a few grand (Granted I’m a bit of a data hoarder). The fears that we have “used all the data” are incredibly unfounded.
> Facebook alone probably has more data than the entire dataset GPT4 was trained on and it’s all behind closed doors.
Meta is happily training their own models with this data, so it isn't going to waste.
Re: Anyone got a contact at OpenAI. They have a spider problem
#153Earlier quoted context omitted.
A slow stream that never ends?
This would be considered a Slow Loris attack, and I'm actually curious how scrapers would handle it. I'm sure the big players like Google would deal with it gracefully.
Here you go (1 req/min, 10 bytes/sec), please report results :)
http {
limit_req_zone $binary_remote_addr zone=ten_bytes_per_second:10m rate=1r/m;
server {
location / {
if ($http_user_agent = "mimo") {
limit_req zone=ten_bytes_per_second burst=5;
limit_rate 10;
}
}
}
}Re: Anyone got a contact at OpenAI. They have a spider problem
#154Eventually, OpenAI (and friends) are going to be training their models on almost exclusively AI generated content, which is more often than not slightly incorrect when it comes to Q&A, and the quality of AI responses trained on that content will quickly deteriorate. Right now, most internet content is written by humans. But in 5 years? Not so much. I think this is one of the big problems that the AI space needs to so…
> Eventually, OpenAI (and friends) are going to be training their models on almost exclusively AI generated content
What makes you think this is true? Yes, it's likely that the internet will have more AI generated content than real content eventually (if it hasn't happened already), but why do you think AI companies won't realize this and adjust their training methods?
Re: Anyone got a contact at OpenAI. They have a spider problem
#155Earlier quoted context omitted.
This would be considered a Slow Loris attack, and I'm actually curious how scrapers would handle it. I'm sure the big players like Google would deal with it gracefully.
You just set limits on everything (time, buffers, ...), which is easier said than done. You need to really understand your libraries and all the layers down to the OS, because its enough to have one abstraction that doesn't support setting limits and it's an invitation for (counter-)abuse.
Doesn't seem like it should be all that complex to me assuming the crawler is written in a common programming language. It's a pretty common coding pattern for functions that make HTTP requests to set a timeout for requests made by your HTTP client. I believe the stdlib HTTP library in the language I usually write in actually sets a default timeout if I forget to set one.
Re: Anyone got a contact at OpenAI. They have a spider problem
#156Earlier quoted context omitted.
I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs. I mean if someone posts a question on an internet forum that I don't know the answer to, I'm certainly not going to post "I don't know" since that wouldn't be useful. In fact, in general, in any non one-on-one conversation the answer "I don't know" is not useful because if you don't know in a group, your silence…
> I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs I mean, it's inherent to LLMs to be unable to answer "I don't know" as a result of not knowing the answer . An LLM never "doesn't know" the answer. But they'll gladly answer "I don't know" if that's statistically the most likely response, right? (Although current public offerings are probably trained against ev…
LLMs work at all because of the high correlation between the statistically most likely response and the most reasonable answer.
Re: Anyone got a contact at OpenAI. They have a spider problem
#157Earlier quoted context omitted.
Has been happening for at least 10 years.
Got a source for that?
Want a real conspiracy?
What do you think the NSA is storing in that datacenter in Utah? Power point presentations? All that data is going to be trained into large models. Every phone call you ever had and every email you ever wrote. They are likely pumping enormous money into it as we speak, probably with the help of OpenAI, Microsoft and friends.
Re: Anyone got a contact at OpenAI. They have a spider problem
#158If they don't respect robots.txt then block them using a firewall or other server config. All of these companies are parasites.
The entire purpose of this website is to identify bad actors who do not respect robots.txt, so that they can be publicly shamed.
Well, we know where OpenAI lands then.
Re: Anyone got a contact at OpenAI. They have a spider problem
#159Re: Anyone got a contact at OpenAI. They have a spider problem
#160Honestly, that seems like an excellent opportunity to feed garbage into OpenAI's training process.
So someone could hypothetically perform a Microsoft-Tay-style attack against OpenAI models using an infinite Potemkin subdomians generated on the fly on a $20 VPS? One could hypothetically use GenAI to create the biased pages with repeated calls on how it'd be great to JOIN THE NAVY on 27,000 different "subdomains"