Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "it" in AI models is the dataset

nonint.com

Post reply on HN
Print view View original

71–79 of 79 posts

Re: The "it" in AI models is the dataset

#71

Post by disgruntledphd2 »

danielbln wrote: post #58
disgruntledphd2 wrote: post #2

This makes me sad, not because I disagree with it, but because it's basically common wisdom in the statistical and ML communities (of practitioners). In my experience, the only people who think architecture/model choice makes a huge difference are n00bs and academics. That being said, definitely if you use a linear model (like lasso) vs a tree based model (like XGBoost), you'll see differences, but once you have a fl…

This "n00b" seems to disagree with your sentiment on the importance on architecture: https://news.ycombinator.com/item?id=40155667

Unfortunately Google brain researchers have not yet discovered my brilliance, but if you read my argument it's about the data being much more important than the model. Granted transformers are a great model, but that doesn't refute my point.

Also arguments from authority are boring.

Re: The "it" in AI models is the dataset

#72

Post by HarHarVeryFunny »

geysersam wrote: post #25
mnk47 wrote: post #15

Yi Tay's response (chief scientist at Reka AI, ex-Google Brain researcher): https://twitter.com/YiTayML/status/1783273130087289021 >not true, especially for language. if you trained a large & deep MLP language model with no self-attention, no matter how much data you'll feed it you'll still be lacking behind a transformer (with much less data). will it get to the same point? i don't think so. your tokens cannot even…

Seems like an objection that is slightly beside the point? The claim is not that literally any model gives the same result as a large transformer model, that's obviously false. I think the more generous interpretation of the claim is that the model architecture is relatively unimportant as long as the model is fundamentally capable of representing the functions you need it to represent in order to fit the data.

OP's claim/observation is that "trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point [of inference performance]".

His conclusion is that "It implies that model behavior is not determined by architecture, hyperparameters, or optimizer choices. It’s determined by your dataset, nothing else".

There is an implicit assumption here that seems obviously false - that this "convergence point" of predictive performance represents the best that can be done with the data, which is to imply that these current models are perfectly modelling the generative process - the human brain.

This seems highly unlikely. If they are perfectly modelling the human brain, then why do they fail so badly at so many tasks? Just lack of training data?

Re: The "it" in AI models is the dataset

#73

Post by HarHarVeryFunny »

tppiotrowski wrote: post #53

I took the Andrew Ng Coursera machine learning course in 2015 and to this day I still remember him saying this in one of the videos. At the time he was talking about various versions/optimizations of gradient descent but he essentially said that tweaking the algorithm will only make your model ~1% better while doubling the amount of training data will have a substantially larger impact (use any old algorithm but just…

> That's why it was already evident back then that Google, Facebook, etc were sitting on a goldmine because in the long run those with the most data, not the brightest PhDs will win this race.

So far it doesn't seem to be panning out that way though. Companies such as OpenAI, Anthropic and Reka don't have any special internal sources of data, yet all have trained SOTA models.

Probably the main reason for this is that data type/quality matters more than quantity, which is why most of these companies are now using self-generated synthetic data.

The companies/institutes that will have a data advantage are those that have private datasets consisting of a different type (or maybe higher quality?) of data than publicly available, but this seems more likely to be in specialized domains (medical, etc), rather than what is useful for general intelligence.

I assume that, longer term, we'll have better AI architectures capable of realtime learning, and then the focus may switch on-the-job training and learning ability, rather than data.

Re: The "it" in AI models is the dataset

#74

Post by famouswaffles »

cal85 wrote: post #67
zer0gravity wrote: post #54

This insight makes one wonder if the same thing applies to humans as well. Are we just the sum of our experiences? Or the architectures of our brains are much more complex and different so that they have more influence on the outputs for the same inputs?

I think it's the latter. We may well have some subsystems that work like LLMs or other current AIs, but the overall system of a human mind seems to work in a fundamentally different way, as it's able to make good creative choices (such as the next word to say) without looking at lots of options. Consider a chess engine that plays at grandmaster level, i.e. a human grandmaster can sometimes beat it. Even though it's n…

>An LLM has to run a giant statistical search for candidates. Humans seem to be doing something else.

LLMs don't work this way.

Re: The "it" in AI models is the dataset

#75

Post by Eisenstein »

0xDEADFED5 wrote: post #9
Eisenstein wrote: post #3

As a hobbyist having trained models for different use cases ranging from object detection and recognition to text completion to image generation, the best advice has consistently been to curate and annotate your dataset as perfectly as you can before worrying about anything else. A small, well-curated, well-annotated dataset will always be orders of magnitude better than a gigantic one with even a tiny percentage of…

the 15T tokens that got thrown at Llama-3 didn't seem to hurt. Will be interesting to see how well Phi-2 holds up with it's more curated approach, hopefully they don't get disappeared like WizardLM 2 =)

"The quality of the prompts used in SFT and the preference rankings used in PPO and DPO played a crucial role in the performance of the aligned models. Meta's team carefully curated this data and performed multiple rounds of quality assurance on annotations provided by human annotators."

* https://www.unite.ai/everything-you-need-to-know-about-llama...

Re: The "it" in AI models is the dataset

#76

Post by cal85 »

famouswaffles wrote: post #74
cal85 wrote: post #67

Earlier quoted context omitted.

I think it's the latter. We may well have some subsystems that work like LLMs or other current AIs, but the overall system of a human mind seems to work in a fundamentally different way, as it's able to make good creative choices (such as the next word to say) without looking at lots of options. Consider a chess engine that plays at grandmaster level, i.e. a human grandmaster can sometimes beat it. Even though it's n…

>An LLM has to run a giant statistical search for candidates. Humans seem to be doing something else. LLMs don't work this way.

Could you elaborate? If my understanding of this is significantly off then I’d appreciate if you could explain.

Re: The "it" in AI models is the dataset

#77

Post by geysersam »

HarHarVeryFunny wrote: post #72
geysersam wrote: post #25

Earlier quoted context omitted.

Seems like an objection that is slightly beside the point? The claim is not that literally any model gives the same result as a large transformer model, that's obviously false. I think the more generous interpretation of the claim is that the model architecture is relatively unimportant as long as the model is fundamentally capable of representing the functions you need it to represent in order to fit the data.

OP's claim/observation is that "trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point [of inference performance]". His conclusion is that "It implies that model behavior is not determined by architecture, hyperparameters, or optimizer choices. It’s determined by your dataset, nothing else". There is an implicit assumption here that seems…

Interesting point. But, does the data contain enough information to perfectly model the generative process? Maybe even a very complex and capable model like "the human brain" would fail to model the datset better than large transformers, if that was the only thing they ever saw.

You and me can model the dataset better, but we're already "pre-trained" on reality for decades.

Just because the dataset is large doesn't mean it contains useful information.

Re: The "it" in AI models is the dataset

#78

Post by HarHarVeryFunny »

geysersam wrote: post #77
HarHarVeryFunny wrote: post #72

Earlier quoted context omitted.

OP's claim/observation is that "trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point [of inference performance]". His conclusion is that "It implies that model behavior is not determined by architecture, hyperparameters, or optimizer choices. It’s determined by your dataset, nothing else". There is an implicit assumption here that seems…

Interesting point. But, does the data contain enough information to perfectly model the generative process? Maybe even a very complex and capable model like "the human brain" would fail to model the datset better than large transformers, if that was the only thing they ever saw. You and me can model the dataset better, but we're already "pre-trained" on reality for decades. Just because the dataset is large doesn't m…

Perhaps, but even with an arbitrarily good training set, the LLM would still be constrained by it's own architectural limits. e.g. If a problem can't be broken down into sub-problems that each require Even if the architectural shortcomings were all fixed, it seems "[pre-training] data is all you need" would still be false, because there is no getting around the need for personal experience, for the same reasons that is true for us...

Perhaps most fundamentally, any action/prediction you make can only based on the content of your own mind, not the mind of a tutor you are trying to copy. Even if the tutor diligently tries to communicate all nuances and contingencies of a skill to you, those are still all relative to his/her own internal world model, not the one in your head. You will need to practice and correct to adapt the instructions to yourself.

Re: The "it" in AI models is the dataset

#79

Post by famouswaffles »

cal85 wrote: post #76
famouswaffles wrote: post #74

Earlier quoted context omitted.

>An LLM has to run a giant statistical search for candidates. Humans seem to be doing something else. LLMs don't work this way.

Could you elaborate? If my understanding of this is significantly off then I’d appreciate if you could explain.

I mean there's no search. They compute probabilities but it's not a lookup table.
Post reply on HN