Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "it" in AI models is the dataset

nonint.com

Post reply on HN
Print view View original

41–50 of 79 posts

Re: The "it" in AI models is the dataset

#41

Post by infecto »

sevagh wrote: post #39
disgruntledphd2 wrote: post #2

This makes me sad, not because I disagree with it, but because it's basically common wisdom in the statistical and ML communities (of practitioners). In my experience, the only people who think architecture/model choice makes a huge difference are n00bs and academics. That being said, definitely if you use a linear model (like lasso) vs a tree based model (like XGBoost), you'll see differences, but once you have a fl…

>In my experience, the only people who think architecture/model choice makes a huge difference are n00bs and academics. There are countless competitions, etc. on Kaggle, AICrowd, or other platforms with an enforced standardized data set. Every entrant uses the same data set and there's a huge difference between the best and worst submissions.

> ...on Kagi,....

Did you mean https://www.kaggle.com/?

Re: The "it" in AI models is the dataset

#42

Post by queuebert »

bilsbie wrote: post #31

Has anyone tried removing an entire concept from a dataset and seeing if the LLM can reason its way into the concept? I think that would be a really cool experiment. There are probably some really good candidate concepts that just take a small leap of reasoning to reach. But off the top of my head maybe multiplication? Or the concept of zero. Maybe the wheel? Edit: if anyone is interesting in doing this kind of stuff…

I want to see an LLM that generates answers without the letter 'e', like the novel Gadsby by Ernest Vincent Wright.

Re: The "it" in AI models is the dataset

#43

Post by sevagh »

infecto wrote: post #41
sevagh wrote: post #39

Earlier quoted context omitted.

>In my experience, the only people who think architecture/model choice makes a huge difference are n00bs and academics. There are countless competitions, etc. on Kaggle, AICrowd, or other platforms with an enforced standardized data set. Every entrant uses the same data set and there's a huge difference between the best and worst submissions.

> ...on Kagi,.... Did you mean https://www.kaggle.com/ ?

Yes, thanks.

Re: The "it" in AI models is the dataset

#44

Post by 4death4 »

mnk47 wrote: post #15

Yi Tay's response (chief scientist at Reka AI, ex-Google Brain researcher): https://twitter.com/YiTayML/status/1783273130087289021 >not true, especially for language. if you trained a large & deep MLP language model with no self-attention, no matter how much data you'll feed it you'll still be lacking behind a transformer (with much less data). will it get to the same point? i don't think so. your tokens cannot even…

MLP is a universal approximator, so there’s definitely a configuration that can match an attention mechanism. Whether or not it’d be feasible to train is another question.

Re: The "it" in AI models is the dataset

#46

Post by ttpphd »

rambambram wrote: post #17

> It is a giant pain in the ass but you have to spend the time sitting in front of the screen going through the data and removing things and tagging things and making sure that the details are right. This is really what makes the good models good and the rest mediocre. In some other comment I read this. Sounds very much like a curation thing. And now I'm wondering; isn't this part already covered by a lot of human be…

Yeah when you use OpenAI you are giving them free labor for data curation.

Re: The "it" in AI models is the dataset

#48

Post by teekert »

omnicognate wrote: post #27
teekert wrote: post #20

I don’t get this: “What that means is not only that they learn what it means to be a dog or a cat, …“ We don’t have any dataset of dog or cat experience right? OP probably means that he models learns wat a dog or cat is, right? I find the whole piece somewhat vague btw. No real insights if you ask me. Sure if all you put in is a dataset, that should be all you get out. What’s surprising (worth HN) here?

> OP probably means that he models learns wat a dog or cat is, right? Yes, "What it means to be" does appear to be meant that way and it didn't occur to me to interpret it the other way. > Sure if all you put in is a dataset, that should be all you get out. What's surprising (worth HN) here? You put in a particular choice of nn architecture as well as the dataset. The insight (to the extent that it is insightful, and…

Ok the first thing must be just my non-native speaker mind then.

The second: still fills like Duh. It’s what these models are meant to do right? Form an internal representation of the relations hidden in the data. It’s what complex systems are, they hold models of reality and use those to predict. That is in fact what Claude Shannon meant with his definition of information. Idk maybe I’m getting it wrong.

Re: The "it" in AI models is the dataset

#49

Post by empath-nirvana »

segmondy wrote: post #37
ahartmetz wrote: post #21

Earlier quoted context omitted.

Well, both can be true if you interpret the "it" as "the secret sauce / competitive advantage". A good architecture is a necessary but not sufficient condition for success, but everybody uses more or less the same currently, so data makes the difference. Until the next improvement in architecture.

I do argue that the IT is the architecture. We have pretty much had all the data that these LLMs were trained on for a long time. The game changer was the architecture not the data. Unless of course you are on the code is data camp ;).

Probably the "it" is whatever one model has that other models don't have. When everyone is using the same architecture, then the data makes the difference. If everyone has the same data, then the architecture makes the difference.

It sounds pretty obvious to say that the difference is whatever is different, but isn't that literally what both sides of this argument are saying?

edit: I do think that what the original linked essay is saying is slightly subtler than that, which is that _given_ that everyone is using the same transformer architecture, the exact hyperparameters and fine tuning that is done matters a lot less than the data set does.

Re: The "it" in AI models is the dataset

#50

Post by HarHarVeryFunny »

4death4 wrote: post #44
mnk47 wrote: post #15

Yi Tay's response (chief scientist at Reka AI, ex-Google Brain researcher): https://twitter.com/YiTayML/status/1783273130087289021 >not true, especially for language. if you trained a large & deep MLP language model with no self-attention, no matter how much data you'll feed it you'll still be lacking behind a transformer (with much less data). will it get to the same point? i don't think so. your tokens cannot even…

MLP is a universal approximator, so there’s definitely a configuration that can match an attention mechanism. Whether or not it’d be feasible to train is another question.

Not sure about feasible, but certainly not efficient.

I think this MLP universal approximator notion is similar to a Turing machine being a universal computation device. Correct, but practically useless.

I don't think Sutton's bitter lesson is going to result in everything being an MLP. You want the most scalable architecture, which an MLP certainly is not.

Post reply on HN