DeepMind’s New Language Model, Chinchilla
marktechpost.com
DeepMind’s New Language Model, Chinchilla
1–10 of 145 posts
Re: DeepMind’s New Language Model, Chinchilla
#2If these things get put on specialized hardware for inference with much lower energy costs, the world will never be the same.
Re: DeepMind’s New Language Model, Chinchilla
#3I have to ask, why call it that? I had a chuckle once I saw the name.
Re: DeepMind’s New Language Model, Chinchilla
#4If these things get put on specialized hardware for inference with much lower energy costs, the world will never be the same.
the biggest problem first of all might be the memory requirements given so many parameters. It couldn't be as cheap as a high end computer in the foreseeable future.
Re: DeepMind’s New Language Model, Chinchilla
#5Seems the link is down. Found a decent synopsis/discussion on lesswrong.
https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scalin...
> On March 29th, DeepMind published a paper, "Training Compute-Optimal Large Language Models", that shows that essentially everyone -- OpenAI, DeepMind, Microsoft, etc. -- has been training large language models with a deeply suboptimal use of compute.
> Following the new scaling laws that they propose for the optimal use of compute, DeepMind trains a new, 70-billion parameter model that outperforms much larger language models, including the 175-billion parameter GPT-3 and DeepMind's own 270-billion parameter "Gopher".
Re: DeepMind’s New Language Model, Chinchilla
#6Cached version since the original is down (I'm assuming it's down due to load issues and not due to the author taking it down). https://webcache.googleusercontent.com/search?q=cache:PLSLy9...
Re: DeepMind’s New Language Model, Chinchilla
#7This is exciting if only because as we discover more compute optimal models that out perform the behemoths that have been state of the art it opens up the ability for smaller independent groups to train and release their own versions, more fully democratizing AI. Looking forward to a group like Eluther or Hugging Face releasing a version of this.
Re: DeepMind’s New Language Model, Chinchilla
#8If these things get put on specialized hardware for inference with much lower energy costs, the world will never be the same.
the biggest problem first of all might be the memory requirements given so many parameters. It couldn't be as cheap as a high end computer in the foreseeable future.
There is probably a space-time trade off that needs to be explored in this space. It might be possible to preload the some of the most likely tokens to be selected next into the cache and/or RAM. These are glorified auto-complete algorithms that are poorly understood, as DeepMind's optimizations appear to show. For the English language, it is probable that there are only so many possible grammatically correct selections for the next token, for example.
Re: DeepMind’s New Language Model, Chinchilla
#9I have to ask, why call it that? I had a chuckle once I saw the name.
Large language models have a (recent) history of silly names. BERT, BART, ELMO, RoBERTa, BIGBIRD, PaLM, Megatron etc. Might as well go full nonsense.
Re: DeepMind’s New Language Model, Chinchilla
#10Earlier quoted context omitted.
the biggest problem first of all might be the memory requirements given so many parameters. It couldn't be as cheap as a high end computer in the foreseeable future.
There is probably a space-time trade off that needs to be explored in this space. It might be possible to preload the some of the most likely tokens to be selected next into the cache and/or RAM. These are glorified auto-complete algorithms that are poorly understood, as DeepMind's optimizations appear to show. For the English language, it is probable that there are only so many possible grammatically correct selecti…
Glorified autocomplete? Autocomplete can guess the next word .. sometimes, GPT-3 goes hundreds of words ahead. On generic topics it can be hard to distinguish from human text.
And it can't cache tokens because all tokens are evaluated in the context of all the other tokens, so they don't have the same representations when they reoccur at different positions.