AKA "group think"
The "it" in AI models is the dataset
11–20 of 79 posts
Re: The "it" in AI models is the dataset
#12As a hobbyist having trained models for different use cases ranging from object detection and recognition to text completion to image generation, the best advice has consistently been to curate and annotate your dataset as perfectly as you can before worrying about anything else. A small, well-curated, well-annotated dataset will always be orders of magnitude better than a gigantic one with even a tiny percentage of…
This is where software developers have a huge role to play: build software that invites user experiences that label as part of the user flow
Re: The "it" in AI models is the dataset
#13This makes me sad, not because I disagree with it, but because it's basically common wisdom in the statistical and ML communities (of practitioners). In my experience, the only people who think architecture/model choice makes a huge difference are n00bs and academics. That being said, definitely if you use a linear model (like lasso) vs a tree based model (like XGBoost), you'll see differences, but once you have a fl…
Why does it make you sad? It seems intuitiv and simple. And in reality of course the optimisation part is not trivial. What would we better if the "it" was more complicated?
It used to be that people would get into these fields thinking ML would need specifically human insights, deep thinking, and philosophical insights about the nature of consciousness.
You would get into natural language modelling because you had a deep love of language. Because you think you're close to figuring language out in a systematic way, with just a few years more study.
There's a certain sadness, I think, in the revelation that the robots don't need the expertise of humanity's greatest experts and masters, they just need us to click all the squares that contain a motorcycle.
Re: The "it" in AI models is the dataset
#14Earlier quoted context omitted.
Why does it make you sad? It seems intuitiv and simple. And in reality of course the optimisation part is not trivial. What would we better if the "it" was more complicated?
It used to be that people would get into these fields thinking ML would need specifically human insights, deep thinking, and philosophical insights about the nature of consciousness. You would get into natural language modelling because you had a deep love of language. Because you think you're close to figuring language out in a systematic way, with just a few years more study. There's a certain sadness, I think, in…
How do you know? We’re not there yet.
Re: The "it" in AI models is the dataset
#15Yi Tay's response (chief scientist at Reka AI, ex-Google Brain researcher): https://twitter.com/YiTayML/status/1783273130087289021
>not true, especially for language. if you trained a large & deep MLP language model with no self-attention, no matter how much data you'll feed it you'll still be lacking behind a transformer (with much less data). will it get to the same point? i don't think so. your tokens cannot even see each other in a raw MLP.
>on the other hand, tiny tweaks to transformers may not matter as much as data/compute. sure. but it's also not very accurate to say "architecture research" does not matter and "makes no difference". i hear this a lot about how people use this to justify not innovating at the architecture level.
>the truth is the community stands on the shoulder of giants of all the arch research that have been done to push the transformer to this state today.
>architecture research matters. many people just take it for granted these days.
Re: The "it" in AI models is the dataset
#16This makes me sad, not because I disagree with it, but because it's basically common wisdom in the statistical and ML communities (of practitioners). In my experience, the only people who think architecture/model choice makes a huge difference are n00bs and academics. That being said, definitely if you use a linear model (like lasso) vs a tree based model (like XGBoost), you'll see differences, but once you have a fl…
Why does it make you sad? It seems intuitiv and simple. And in reality of course the optimisation part is not trivial. What would we better if the "it" was more complicated?
Because of the way it's presented, as if it's some vast new discovery that OpenAI have made, rather than common wisdom.
It makes me sad when people rediscover things (with massive compute in this case), that were already known.
It's very much spend a year in the lab to save an hour in the library.
Re: The "it" in AI models is the dataset
#17> It is a giant pain in the ass but you have to spend the time sitting in front of the screen going through the data and removing things and tagging things and making sure that the details are right. This is really what makes the good models good and the rest mediocre.
In some other comment I read this. Sounds very much like a curation thing. And now I'm wondering; isn't this part already covered by a lot of human beings now interacting with ChatGPT and the like?
My uneducated guess is that a company can scrape the whole world wide web and also have all the low quality content that comes with it, but then strengthen/curate their data and/or model by having it interact with humans? You give this thing a prompt, it comes up with some obvious nonsense, and then you as a human correct this by 'chatting' with it?
Re: The "it" in AI models is the dataset
#18Earlier quoted context omitted.
Why does it make you sad? It seems intuitiv and simple. And in reality of course the optimisation part is not trivial. What would we better if the "it" was more complicated?
It used to be that people would get into these fields thinking ML would need specifically human insights, deep thinking, and philosophical insights about the nature of consciousness. You would get into natural language modelling because you had a deep love of language. Because you think you're close to figuring language out in a systematic way, with just a few years more study. There's a certain sadness, I think, in…
This is 100% not why I am sad, see my other reply for information.
As an aside, it's wild how people put their own spin onto what I said.
Obviously I should have been clearer :shrug:.
Re: The "it" in AI models is the dataset
#19Yes, and it's what people seem to ignore when they talk about dethroning GPT4 as the top LLM. It's good data expressly developed for training the behaviors they want that keeps them ahead, all the other stuff (other training and filtering web data) has much less of an impact.
See also "You won't train a better model from your desk: https://news.ycombinator.com/item?id=40155715
Re: The "it" in AI models is the dataset
#20I don’t get this: “What that means is not only that they learn what it means to be a dog or a cat, …“
We don’t have any dataset of dog or cat experience right? OP probably means that he models learns wat a dog or cat is, right?
I find the whole piece somewhat vague btw. No real insights if you ask me. Sure if all you put in is a dataset, that should be all you get out. What’s surprising (worth HN) here?