We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.
Many in the AI field think the bigger-is-better approach is running out of road
11–20 of 354 posts
Re: Many in the AI field think the bigger-is-better approach is running out of road
#12Another recent (but not called out in this article) is the "Textbooks Are All You Need" paper [1]; the results seem to suggest that careful curation and curriculums of training data can significantly improve model capabilities (when training domain specific, smaller models). Claiming a 10x smaller model can outperform competitors. (Eg. phi-1 vs. starcoder)
Re: Many in the AI field think the bigger-is-better approach is running out of road
#13We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.
Logistic Regression is simple to implement, supports binomial, multinomial, and ordinal classification, and is a key layer in NN, as it's often used as an actuator that sorts a probability into a discrete category. Very good for specialized problems, and easily trainable to sort unknown or nonsensical inputs into a noncategory.
Linear Regression is great for projections, and can even be fit to time series data using lagging.
Re: Many in the AI field think the bigger-is-better approach is running out of road
#14Data requirements are overstated - you can train on longer and longer sequences and I am pretty sure most organizations are still using the “show the model the data only once“ approach which is just wasteful. Compute challenges are more real, but we are seeing for the first time huge amounts of global capital being allocated to solve specifically these problems, so I am curious what fruit that will bear in a few year…
Yep. We are in the very early innings of capital being deployed to all this.
Re: Many in the AI field think the bigger-is-better approach is running out of road
#15Another recent (but not called out in this article) is the "Textbooks Are All You Need" paper [1]; the results seem to suggest that careful curation and curriculums of training data can significantly improve model capabilities (when training domain specific, smaller models). Claiming a 10x smaller model can outperform competitors. (Eg. phi-1 vs. starcoder) [1] https://arxiv.org/abs/2306.11644
I’ve still wondered if anyone has tried training with a large dataset of published books, like something from library genesis, or in the case of google using the full text from google books. There’s all this talk of finding quality text and I’ve not heard of text from print books being a major source beyond this textbooks paper?
Re: Many in the AI field think the bigger-is-better approach is running out of road
#16To the outsider it might seem that the only thing we've been doing is scaling up the neural networks, but that's not true. A lot of innovation and changes happened, some enabled us to scale up more and others just improved performance. I am quite confident that innovation will continue.
Re: Many in the AI field think the bigger-is-better approach is running out of road
#17Another recent (but not called out in this article) is the "Textbooks Are All You Need" paper [1]; the results seem to suggest that careful curation and curriculums of training data can significantly improve model capabilities (when training domain specific, smaller models). Claiming a 10x smaller model can outperform competitors. (Eg. phi-1 vs. starcoder) [1] https://arxiv.org/abs/2306.11644
I’ve still wondered if anyone has tried training with a large dataset of published books, like something from library genesis, or in the case of google using the full text from google books. There’s all this talk of finding quality text and I’ve not heard of text from print books being a major source beyond this textbooks paper?
That’s how OpenAI was (is) doing it. Books downloaded from the Internet is a part of the dataset as per GPT3 model card. Right to read.
Re: Many in the AI field think the bigger-is-better approach is running out of road
#18Data requirements are overstated - you can train on longer and longer sequences and I am pretty sure most organizations are still using the “show the model the data only once“ approach which is just wasteful. Compute challenges are more real, but we are seeing for the first time huge amounts of global capital being allocated to solve specifically these problems, so I am curious what fruit that will bear in a few year…
> using the “show the model the data only once“ approach which is just wasteful.
According to the InstructGPT paper, that is not the case, showing the data multiple times results in overfitting.
Re: Many in the AI field think the bigger-is-better approach is running out of road
#19We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.
Bingo. I've been beating this drum since the initial GPT-3 awe.. The future of AI is bespoke, purpose-driven models trained on a combination of public and (importantly) proprietary data.
Data is still king.
Re: Many in the AI field think the bigger-is-better approach is running out of road
#20To the outsider it might seem that the only thing we've been doing is scaling up the neural networks, but that's not true. A lot of innovation and changes happened, some enabled us to scale up more and others just improved performance. I am quite confident that innovation will continue.
Your statement is entirely fair, but the actual title is "The bigger-is-better approach to AI is running out of road". They are actually saying what you are saying, but your comment seems to contest the article.