Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The "it" in AI models is the dataset

nonint.com

Post reply on HN
Print view View original

21–30 of 79 posts

Re: The "it" in AI models is the dataset

#21

Post by ahartmetz »

mnk47 wrote: post #15

Yi Tay's response (chief scientist at Reka AI, ex-Google Brain researcher): https://twitter.com/YiTayML/status/1783273130087289021 >not true, especially for language. if you trained a large & deep MLP language model with no self-attention, no matter how much data you'll feed it you'll still be lacking behind a transformer (with much less data). will it get to the same point? i don't think so. your tokens cannot even…

Well, both can be true if you interpret the "it" as "the secret sauce / competitive advantage". A good architecture is a necessary but not sufficient condition for success, but everybody uses more or less the same currently, so data makes the difference. Until the next improvement in architecture.

Re: The "it" in AI models is the dataset

#24

Post by xanderlewis »

disgruntledphd2 wrote: post #18
michaelt wrote: post #13

Earlier quoted context omitted.

It used to be that people would get into these fields thinking ML would need specifically human insights, deep thinking, and philosophical insights about the nature of consciousness. You would get into natural language modelling because you had a deep love of language. Because you think you're close to figuring language out in a systematic way, with just a few years more study. There's a certain sadness, I think, in…

This is 100% not why I am sad, see my other reply for information. As an aside, it's wild how people put their own spin onto what I said. Obviously I should have been clearer :shrug:.

Well, you have to forgive some for making assumptions based on your choice of username…

Re: The "it" in AI models is the dataset

#25

Post by geysersam »

mnk47 wrote: post #15

Yi Tay's response (chief scientist at Reka AI, ex-Google Brain researcher): https://twitter.com/YiTayML/status/1783273130087289021 >not true, especially for language. if you trained a large & deep MLP language model with no self-attention, no matter how much data you'll feed it you'll still be lacking behind a transformer (with much less data). will it get to the same point? i don't think so. your tokens cannot even…

Seems like an objection that is slightly beside the point? The claim is not that literally any model gives the same result as a large transformer model, that's obviously false. I think the more generous interpretation of the claim is that the model architecture is relatively unimportant as long as the model is fundamentally capable of representing the functions you need it to represent in order to fit the data.

Re: The "it" in AI models is the dataset

#26

Post by disgruntledphd2 »

xanderlewis wrote: post #24
disgruntledphd2 wrote: post #18

Earlier quoted context omitted.

This is 100% not why I am sad, see my other reply for information. As an aside, it's wild how people put their own spin onto what I said. Obviously I should have been clearer :shrug:.

Well, you have to forgive some for making assumptions based on your choice of username…

Fair, I'm just generally disgruntled to be fair, the PhD was just the name of my soon abandoned blog.

Re: The "it" in AI models is the dataset

#27

Post by omnicognate »

teekert wrote: post #20

I don’t get this: “What that means is not only that they learn what it means to be a dog or a cat, …“ We don’t have any dataset of dog or cat experience right? OP probably means that he models learns wat a dog or cat is, right? I find the whole piece somewhat vague btw. No real insights if you ask me. Sure if all you put in is a dataset, that should be all you get out. What’s surprising (worth HN) here?

> OP probably means that he models learns wat a dog or cat is, right?

Yes, "What it means to be" does appear to be meant that way and it didn't occur to me to interpret it the other way.

> Sure if all you put in is a dataset, that should be all you get out. What's surprising (worth HN) here?

You put in a particular choice of nn architecture as well as the dataset. The insight (to the extent that it is insightful, and true) is that the architecture doesn't affect the results you get much compared to the dataset.

Re: The "it" in AI models is the dataset

#28

Post by omnicognate »

mnk47 wrote: post #15

Yi Tay's response (chief scientist at Reka AI, ex-Google Brain researcher): https://twitter.com/YiTayML/status/1783273130087289021 >not true, especially for language. if you trained a large & deep MLP language model with no self-attention, no matter how much data you'll feed it you'll still be lacking behind a transformer (with much less data). will it get to the same point? i don't think so. your tokens cannot even…

Machine learning insights from e e cummings.

Re: The "it" in AI models is the dataset

#29

Post by Hendrikto »

rambambram wrote: post #17

> It is a giant pain in the ass but you have to spend the time sitting in front of the screen going through the data and removing things and tagging things and making sure that the details are right. This is really what makes the good models good and the rest mediocre. In some other comment I read this. Sounds very much like a curation thing. And now I'm wondering; isn't this part already covered by a lot of human be…

People typically ask LLMs about things they DON‘T know about or understand. So they are not qualified to assess the validity of their answers. Which is exactly why hallucination is such a big problem.

Re: The "it" in AI models is the dataset

#30

Post by nkozyra »

ahartmetz wrote: post #21
mnk47 wrote: post #15

Yi Tay's response (chief scientist at Reka AI, ex-Google Brain researcher): https://twitter.com/YiTayML/status/1783273130087289021 >not true, especially for language. if you trained a large & deep MLP language model with no self-attention, no matter how much data you'll feed it you'll still be lacking behind a transformer (with much less data). will it get to the same point? i don't think so. your tokens cannot even…

Well, both can be true if you interpret the "it" as "the secret sauce / competitive advantage". A good architecture is a necessary but not sufficient condition for success, but everybody uses more or less the same currently, so data makes the difference. Until the next improvement in architecture.

Or until we run out of data that actually differentiates the models
Post reply on HN