The "it" in AI models is the dataset
nonint.com
The "it" in AI models is the dataset
1–10 of 79 posts
Re: The "it" in AI models is the dataset
#2This makes me sad, not because I disagree with it, but because it's basically common wisdom in the statistical and ML communities (of practitioners). In my experience, the only people who think architecture/model choice makes a huge difference are n00bs and academics.
That being said, definitely if you use a linear model (like lasso) vs a tree based model (like XGBoost), you'll see differences, but once you have a flexible enough model and a lot of data, training time and inference complexity tend to become better ways to make a model choice.
Re: The "it" in AI models is the dataset
#3As a hobbyist having trained models for different use cases ranging from object detection and recognition to text completion to image generation, the best advice has consistently been to curate and annotate your dataset as perfectly as you can before worrying about anything else.
A small, well-curated, well-annotated dataset will always be orders of magnitude better than a gigantic one with even a tiny percentage of mislabeled features or bad/wrong data. Hyperparameters and such can be fiddled with once you know you are on the right the track and in the scheme of things are relatively minor for most purposes.
Of course, this advice gets routinely ignored as people spend countless hours fussing over how to set certain flags and grabbing as much data as possible, then carelessly throwing it all together and training it. Then, wondering why the model does things they don't want, they go back to messing with the parameters again.
It is a giant pain in the ass but you have to spend the time sitting in front of the screen going through the data and removing things and tagging things and making sure that the details are right. This is really what makes the good models good and the rest mediocre.
Re: The "it" in AI models is the dataset
#4This makes me sad, not because I disagree with it, but because it's basically common wisdom in the statistical and ML communities (of practitioners). In my experience, the only people who think architecture/model choice makes a huge difference are n00bs and academics. That being said, definitely if you use a linear model (like lasso) vs a tree based model (like XGBoost), you'll see differences, but once you have a fl…
Why does it make you sad? It seems intuitiv and simple. And in reality of course the optimisation part is not trivial. What would we better if the "it" was more complicated?
Re: The "it" in AI models is the dataset
#5Alon Halevy, Peter Norvig, and Fernando Pereira (2009): The Unreasonable
Effectiveness of Data
https://static.googleusercontent.com/media/research.google.c...
Re: The "it" in AI models is the dataset
#6The only thing this glosses over is RL. I guess you can see agents interacting in environments as a type of "dataset", but it _feels_ different.
Re: The "it" in AI models is the dataset
#7Alon Halevy, Peter Norvig, and Fernando Pereira (2009): The Unreasonable Effectiveness of Data https://static.googleusercontent.com/media/research.google.c...
Rylan Schaeffer (2023): Pretraining on the Test Set Is All You Need
Re: The "it" in AI models is the dataset
#8Is the secret sauce also tied to the generation distribution which can differ from the dataset distribution e.g. RLHF?
Re: The "it" in AI models is the dataset
#9As a hobbyist having trained models for different use cases ranging from object detection and recognition to text completion to image generation, the best advice has consistently been to curate and annotate your dataset as perfectly as you can before worrying about anything else. A small, well-curated, well-annotated dataset will always be orders of magnitude better than a gigantic one with even a tiny percentage of…
the 15T tokens that got thrown at Llama-3 didn't seem to hurt. Will be interesting to see how well Phi-2 holds up with it's more curated approach, hopefully they don't get disappeared like WizardLM 2 =)
Re: The "it" in AI models is the dataset
#10This makes me sad, not because I disagree with it, but because it's basically common wisdom in the statistical and ML communities (of practitioners). In my experience, the only people who think architecture/model choice makes a huge difference are n00bs and academics. That being said, definitely if you use a linear model (like lasso) vs a tree based model (like XGBoost), you'll see differences, but once you have a fl…
Why does it make you sad? It seems intuitiv and simple. And in reality of course the optimisation part is not trivial. What would we better if the "it" was more complicated?
Possibly because being in the business of trying to turn iq edge into money, not data edge into money.