Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anyone got a contact at OpenAI. They have a spider problem

mailman.nanog.org

Post reply on HN
Print view View original

181–190 of 400 posts

Re: Anyone got a contact at OpenAI. They have a spider problem

#181

Post by bevekspldnw »

Eisenstein wrote: post #152
oceanplexian wrote: post #149

Earlier quoted context omitted.

Content you’re allowed and capable of scraping on the Internet is such a small amount of data, not sure why people are acting otherwise. Common crawl alone is only a few hundred TB, I have more content than that on a NAS sitting in my office that I built for a few grand (Granted I’m a bit of a data hoarder). The fears that we have “used all the data” are incredibly unfounded.

> Facebook alone probably has more data than the entire dataset GPT4 was trained on and it’s all behind closed doors. Meta is happily training their own models with this data, so it isn't going to waste.

Not Llama, they’ve been really clear about that. Especially with DMA cross-joining provisions and various privacy requirements it’s really hard for them, same for Google.

However, Microsoft has been flying under the radar. If they gave all Hotmail and O365 data to OpenAI I’d not be surprised in the slightest.

Re: Anyone got a contact at OpenAI. They have a spider problem

#182

Post by loloquwowndueo »

bogwog wrote: post #154
ezekg wrote: post #71

Eventually, OpenAI (and friends) are going to be training their models on almost exclusively AI generated content, which is more often than not slightly incorrect when it comes to Q&A, and the quality of AI responses trained on that content will quickly deteriorate. Right now, most internet content is written by humans. But in 5 years? Not so much. I think this is one of the big problems that the AI space needs to so…

> Eventually, OpenAI (and friends) are going to be training their models on almost exclusively AI generated content What makes you think this is true? Yes, it's likely that the internet will have more AI generated content than real content eventually (if it hasn't happened already), but why do you think AI companies won't realize this and adjust their training methods?

Many AI content detectors have been retired because they are unreliable - AI can’t consistently identify AI-generated content. How would they adjust then?

Re: Anyone got a contact at OpenAI. They have a spider problem

#183

Post by sumtechguy »

danenania wrote: post #134
CuriouslyC wrote: post #122

Earlier quoted context omitted.

A lot of LLM hallucination is because of the internal conflict between alignment for helpfulness and lack of a clear answer. It's much like when someone gets out of their depth in a conversation and dissembles their way through to try and maintain their illusion of competence. In these cases, if you give the LLM explicit permission to tell you that it doesn't know in cases where it's not sure, that will significantly…

"In these cases, if you give the LLM explicit permission to tell you that it doesn't know in cases where it's not sure, that will significantly reduce hallucinations." I've noticed that while this can help to prevent hallucinations, it can also cause it to go way too far in the other direction and start telling you it doesn't know for all kinds of questions it really can answer.

My current favorite one is to ask the time. Then ask it if it is possible for it to give you the time. You get 2 very different answers.

Re: Anyone got a contact at OpenAI. They have a spider problem

#184

Post by choilive »

oceanplexian wrote: post #157
khalladay wrote: post #109

Earlier quoted context omitted.

Got a source for that?

Want a real conspiracy? What do you think the NSA is storing in that datacenter in Utah? Power point presentations? All that data is going to be trained into large models. Every phone call you ever had and every email you ever wrote. They are likely pumping enormous money into it as we speak, probably with the help of OpenAI, Microsoft and friends.

As I understand it, they don't have the capability to essentially PCAP all that data.. and the data wouldn't be that useful since most interesting traffic is encrypted as well. Instead they store the metadata around the traffic. Phone number X made an outgoing call to Y @ timestamp A, call ended at timestamp B, approximate location is Z, etc. Repeat that for internet IP addresses do some analysis and then you can build a pretty interesting web of connections and how they interact.

Re: Anyone got a contact at OpenAI. They have a spider problem

#185

Post by ben_w »

digging wrote: post #164
aidenn0 wrote: post #156

Earlier quoted context omitted.

LLMs work at all because of the high correlation between the statistically most likely response and the most reasonable answer.

That's an explanation of why their answers can be useful, but doesn't relate to their ability to "not know" an answer

I suspect this is going to be a disagreement on the meaning of "to know".

On the same lines as why people argue if a tree falling in a wood where nobody can hear it makes sound because some people implicitly regard sound is the qualia while others regard it as the vibrations in the air.

Re: Anyone got a contact at OpenAI. They have a spider problem

#186

Post by bevekspldnw »

berniedurfee wrote: post #173
ezekg wrote: post #104

Earlier quoted context omitted.

> The end state of training on web text has always been an ouroboros And when other mediums have been saturated with AI? Books, music, radio, podcasts, movies -- what then? Do we need a (curated?) unadulterated stockpile of human content to avoid the enshittification of everything?

Yahoo.com will rise from the ashes.

I mean, you’re not wrong. I’ve been building some unrelated web search tech and have considered just indexing all the sites I can about and making my own “non shit” search engine. Which really isn’t too hard if you want to do say, 10-50 sites. You can fit that on one 4TB nvme drive on a local workstation.

I’m trying to work on monetization for my product now. The “personal Google” idea is really just an accidental byproduct of solving a much harder task. Not sure if people would pay for that alone.

Re: Anyone got a contact at OpenAI. They have a spider problem

#187

Post by ben_w »

ethanbond wrote: post #115
weregiraffe wrote: post #113

Earlier quoted context omitted.

>Do we need a (curated?) unadulterated stockpile of human content to avoid the enshittification of everything? Either that, or a human level AI.

Well no, we need billions of human-level AIs who are experiencing a world as rich and various as the world that the billions of humans inhabit.

Once we've got the first, making a billion is easy.

That said… are content creators collectively (all media, film and books as well as web) a thin tail or a fat tail?

I could easily believe most of the actual culture comes from 10k-100k people today, even if there's, IDK, ten million YouTubers or something (I have a YouTube channel, something like 14 k views over 14 years, this isn't "culturally relevant" scale, and even if it had been most of those views are for algorithmically generated music from 2010 that's a literal Markov chain).

Re: Anyone got a contact at OpenAI. They have a spider problem

#188

Post by bevekspldnw »

oceanplexian wrote: post #149
bevekspldnw wrote: post #77

Earlier quoted context omitted.

The end state of training on web text has always been an ouroboros - primarily because of adtech incentives to produce low quality content at scale to capture micro pennies. The irony of the whole thing is brutal.

Content you’re allowed and capable of scraping on the Internet is such a small amount of data, not sure why people are acting otherwise. Common crawl alone is only a few hundred TB, I have more content than that on a NAS sitting in my office that I built for a few grand (Granted I’m a bit of a data hoarder). The fears that we have “used all the data” are incredibly unfounded.

Gonna say you’re way off there. Once you decompress common crawl and index it for FTS and put it on fast storage you’re in for some serious pain, and that’s before you even put it in your ML pipeline.

Even refined web runs about 2TB once loaded into Postgres with TS vector columns, and that’s a substantially smaller dataset than common crawl.

It’s not just a dumping a to of zip files on your NAS, it’s making the data responsive and usable.

Re: Anyone got a contact at OpenAI. They have a spider problem

#190

Post by Rebelgecko »

aidenn0 wrote: post #86
btown wrote: post #30

This reminds me of how GPT-2/3/J came across https://reddit.com/r/counting , wherein redditors repeatedly post incremental numbers to count to infinity. It considered their usernames, like SolidGoldMagikarp, such common strings on the Internet that, during tokenization, it treated them as top-level tokens of their own. https://www.alignmentforum.org/posts/8viQEp8KBg2QSW4Yc/solid... https://www.lesswrong.com/posts/LAx…

I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs. I mean if someone posts a question on an internet forum that I don't know the answer to, I'm certainly not going to post "I don't know" since that wouldn't be useful. In fact, in general, in any non one-on-one conversation the answer "I don't know" is not useful because if you don't know in a group, your silence…

Reminds me of a joke

Three logicians walk into a bar. The bartender says "what'll it be, three beers?" The first logician says "I don't know". The second logician says "I don't know". The third logician says "Yes".

Post reply on HN