31–40 of 354 posts
Earlier quoted context omitted.
Search has definitely been dead since before LLMs, we just don't have a replacement yet.
I mean more the concept of search, not the current implementation
We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.
The other good use cases are using LLM to turn natural language prompts into API calls to real data.
Earlier quoted context omitted.
Have those reports from George Hotz been confirmed? It seems plausible to me, but also suggests to me that we have further to go by using that parameter budget for depth rather than for width.
It seems consistent with the behavior we see when using GPT4 in chat mode. Every once in a while it will change its answer as it’s generating it, as though it’s switched which model it favors to produce the response. GPT3.5 doesn’t do that.
We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.
It seems that what we need to make a big leap forward is better reasoning. There is a lot of debate between the GPT-4 can/can't reason camps, but I haven't seen anyone try to argue that it reasons particularly well.
We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.
GPT4 is really good at code and you can generally verify hallucination easily. The other good use cases are using LLM to turn natural language prompts into API calls to real data.
Doing this a couple of times gives me 100% accuracy for my use case that involves some level of summarization and reasoning.
Hallucinations are not as big of a deal at all IMO. Not enough that I'll just sit there and wait for models that don't hallucinate.
We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.
Earlier quoted context omitted.
> using the “show the model the data only once“ approach which is just wasteful. According to the InstructGPT paper, that is not the case, showing the data multiple times results in overfitting.
1. You are just referring to fine tuning, I am referring to training the base model. 2. They still saw performance improvements which is why they did train on the data multiple times, you can see in the paper. 3. there was a recent paper demonstrating that reusing data still saw continued improvements in perplexity, i am on my ipad so cannot find it now
Ahh mb! Sorry.
How is the economist qualified to answer this question?
Using this logic, how are they qualified to answer 90% of the questions their articles deal with...
This specific article seems to be reporting on a very technical issue on how to continue to scale LLM. Even scientific papers have a hard time answering those kind of questions because unless in very special circumstances where we can show with a good confidence that there are limitations (P vs NP for instance) the answer will simply be given by the most successful approach.
We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.
Training on something huge like "the internet" is what gives rise to those amazing emergent properties missing in smaller models (including the recent Pi model). And there are only so many datasets that huge.
But its also indeed a waste, as Pi proves.
There probably is some sweet spot (6B-40B?) for specialized, heavily focused models pre trained with high quality general data.