Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anyone got a contact at OpenAI. They have a spider problem

mailman.nanog.org

Post reply on HN
Print view View original

321–330 of 400 posts

Re: Anyone got a contact at OpenAI. They have a spider problem

#321

Post by trescenzi »

ShamelessC wrote: post #298
anonymousDan wrote: post #272

Earlier quoted context omitted.

What has it got to do with deduplication? I'm talking about crafting some kind of alternative (not necessarily duplicate) data. I agree some kind of post data collection cleaning/filtering of the data before training could potentially catch it. But maybe not!

Ah fair enough. The OP here mentioned having highly similar content on each of the many domains.

The funny way to do this would be to use an LLM to generate the content you respond with. Have 2 smallish LLMs talk to each other about topics chosen at random and generate infinite nonsense pages that have a few hundred words.

Re: Anyone got a contact at OpenAI. They have a spider problem

#322

Post by jvanderbot »

realfeel78 wrote: post #313
jvanderbot wrote: post #235

Earlier quoted context omitted.

Which is crazy because there's plenty of good content for kids on Youtube (if you really need a break!). Blippy, Meekah, Seasame Street, even that mind-numbing drivel Cocomelon (which at least got my girls talking/singing really early).

There's actually no such thing as good "content" for kids, sorry.

I get the sentiment, but when reality hits unrealistic parental expectations, things get messy.

If you have to put a show on TV to give some songs to sing along to or to distract them while you're making lunch, I'm not judging you, and I think it's best to put this content on a gradient rather than black and white.

Re: Anyone got a contact at OpenAI. They have a spider problem

#323

Post by robertlagrant »

Rebelgecko wrote: post #190
aidenn0 wrote: post #86

Earlier quoted context omitted.

I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs. I mean if someone posts a question on an internet forum that I don't know the answer to, I'm certainly not going to post "I don't know" since that wouldn't be useful. In fact, in general, in any non one-on-one conversation the answer "I don't know" is not useful because if you don't know in a group, your silence…

Reminds me of a joke Three logicians walk into a bar. The bartender says "what'll it be, three beers?" The first logician says "I don't know". The second logician says "I don't know". The third logician says "Yes".

That sounds like my old neighbour, a professor of logic from the university of science.

Re: Anyone got a contact at OpenAI. They have a spider problem

#324

Post by t0astbread »

KptMarchewa wrote: post #300
mgsouth wrote: post #292

Earlier quoted context omitted.

If, like me, you didn't get the joke at first: Both of the first two logicians wanted a beer; otherwise they would know the answer was "no". The third logician recognizes this, and therefore knows the answer.

Unless one of those wanted two beers. Or 0.5 beer. Or -1 beers. Or 1e9 beers. Or 2147483648 beers.

Hopefully no one asks where the bathroom is.

Re: Anyone got a contact at OpenAI. They have a spider problem

#325

Post by bevekspldnw »

michaelt wrote: post #264
Dylan16807 wrote: post #248

Earlier quoted context omitted.

How important is full text search for training an LLM, compared to a pile of zip files with a gigabyte of text each?

Maybe not full full text search, but you'll generally want to remove the duplicates and suchlike.

my guess is mistral is so good for its size because they are doing some real precise pre ingestion selection and sorting. You need FTS to do that work.

Re: Anyone got a contact at OpenAI. They have a spider problem

#326

Post by mminer237 »

swatcoder wrote: post #131
GaggiX wrote: post #57

Earlier quoted context omitted.

It seems to respect it as the majority of the requests are for the robots.txt.

I'm not sure any publisher means for their robots.txt to be read as: "You're disallowed, but go head and slurp the content anyway so you can look for external links or any indication that maybe you are allowed to digest this material anyway, and then interpret that how you'd like. I trust you to know what's best and I'm sure you kind of get the gist of what I mean here."

How would one know he is disallowed without reading each site?

Re: Anyone got a contact at OpenAI. They have a spider problem

#327

Post by eightysixfour »

ezekg wrote: post #249
eightysixfour wrote: post #223

Earlier quoted context omitted.

You need to separate “content” and “knowledge.” GenAI can create massive amounts of content, but the knowledge you give it to create that content is what matters and why RAG is the most important pattern right now. From “known good” sources of knowledge, we can generate an infinite amount of content. We can add more “known good” knowledge to the model by generating content about that knowledge and training on it. I a…

> We can add more “known good” knowledge to the model by generating content about that knowledge and training on it. That's my entire point -- AI only generates content right now, but it will also be the source of content for training purposes soon. We need a "known good" human knowledge-base, otherwise generative AI will degenerate as AI generated content proliferates. Crawling the web, like in the case of the OP, i…

> Crawling the web, like in the case of the OP, isn't going to work for much longer. And books, video, and music are next.

That is training on content.

The future will have models pre-trained on content and tuned on corpuses of knowledge. The knowledge it is trained on will be a selling point for the model.

Think of it this way - if you want to update the model so it knows the latest news, does it matter if the news was AI generated if it was generated from details of actual events?

Re: Anyone got a contact at OpenAI. They have a spider problem

#328

Post by eightysixfour »

FridgeSeal wrote: post #299
eightysixfour wrote: post #203

Earlier quoted context omitted.

It is already solved. Look at how Microsoft trained Phi - they used existing models to generate synthetic data from textbooks. That allowed them to create a new dataset grounded in “fact” at a far higher quality than common crawl or others. It looks less like an ouroboros and more like a bootstrapping problem.

Is this like, the AI equivalent of “another layer will fix it” that crypto fans used? “It’s ok bro, another model will fix, just please, one more ~layer~ ~agent~ model” It’s all fun and games until you can’t reliably generate your base models anymore, because all your _base_ data is too polluted. Let’s not forget MS has a $10bn stake in the current crop of LLM’s turning out to be as magic as they claim, so I’m sure t…

I mean, you can use Phi now and it outclasses anything else in its size. This isn’t some “it could happen” situation.

Re: Anyone got a contact at OpenAI. They have a spider problem

#329

Post by bigiain »

Rebelgecko wrote: post #190
aidenn0 wrote: post #86

Earlier quoted context omitted.

I wonder how much the source content is the cause of hallucinations rather than anything inherent to LLMs. I mean if someone posts a question on an internet forum that I don't know the answer to, I'm certainly not going to post "I don't know" since that wouldn't be useful. In fact, in general, in any non one-on-one conversation the answer "I don't know" is not useful because if you don't know in a group, your silence…

Reminds me of a joke Three logicians walk into a bar. The bartender says "what'll it be, three beers?" The first logician says "I don't know". The second logician says "I don't know". The third logician says "Yes".

Found the QA engineer:

https://news.ycombinator.com/item?id=25851770

Re: Anyone got a contact at OpenAI. They have a spider problem

#330

Post by latency-guy2 »

Zambyte wrote: post #258
Tubbe wrote: post #246

Earlier quoted context omitted.

(admittedly)

The (allegedly) implies they do know, but to avoid possible litigation they feign ignorance. The CTO of ClosedAI is probably not a complete idiot.

> The CTO of ClosedAI is probably not a complete idiot.

Either way, ignorance is not an excuse despite how people generally react to it.

Post reply on HN