Can't wait for DeepMind to take a stab at outcompeting dall-e.
DeepMind’s New Language Model, Chinchilla
11–20 of 145 posts
Re: DeepMind’s New Language Model, Chinchilla
#12Earlier quoted context omitted.
There is probably a space-time trade off that needs to be explored in this space. It might be possible to preload the some of the most likely tokens to be selected next into the cache and/or RAM. These are glorified auto-complete algorithms that are poorly understood, as DeepMind's optimizations appear to show. For the English language, it is probable that there are only so many possible grammatically correct selecti…
Glorified autocomplete? Autocomplete can guess the next word .. sometimes, GPT-3 goes hundreds of words ahead. On generic topics it can be hard to distinguish from human text. And it can't cache tokens because all tokens are evaluated in the context of all the other tokens, so they don't have the same representations when they reoccur at different positions.
They're evaluated in the context of the last 2^n many tokens, for many models it is 1024, 2048, or 4096 tokens as a scanning window. The tokens (words and sometimes punctuation) are represented by integer values, so the last 2^n many tokens would certainly qualify for storage in a cache. Then next token selection only has so many possible assignable selections in any given language model because of grammatical limitations. This is only one such optimization, there could also be optimizations around the likelihood of certain words to be used given the presence of certain previous tokens, and so on.
But, yes, tokens are chosen one word as a time based on the previous content, similar to earlier auto-completion algorithms.
Re: DeepMind’s New Language Model, Chinchilla
#13If these things get put on specialized hardware for inference with much lower energy costs, the world will never be the same.
Imagine any diffusion-style text-to-image model on specialized ASIC hardware.
Re: DeepMind’s New Language Model, Chinchilla
#14I have to ask, why call it that? I had a chuckle once I saw the name.
Large language models have a (recent) history of silly names. BERT, BART, ELMO, RoBERTa, BIGBIRD, PaLM, Megatron etc. Might as well go full nonsense.
True. I will add that it is customary to justify it by demonstrating it is some sort of acronym or contraction.
Re: DeepMind’s New Language Model, Chinchilla
#15Is there a good reference as to what a "parameter" is in this context? I've looked a few times, but the explanations don't make any sense to me.
Re: DeepMind’s New Language Model, Chinchilla
#16Is there a good reference as to what a "parameter" is in this context? I've looked a few times, but the explanations don't make any sense to me.
You can think of a parameter as a number you can tweak while training. This network has 70B such numbers.
Re: DeepMind’s New Language Model, Chinchilla
#17I have to ask, why call it that? I had a chuckle once I saw the name.
It outperforms the Gopher model
Re: DeepMind’s New Language Model, Chinchilla
#18I have to ask, why call it that? I had a chuckle once I saw the name.
Large language models have a (recent) history of silly names. BERT, BART, ELMO, RoBERTa, BIGBIRD, PaLM, Megatron etc. Might as well go full nonsense.
A touch of irony that cutting edge research on language can’t produce better names.
Re: DeepMind’s New Language Model, Chinchilla
#19Earlier quoted context omitted.
Large language models have a (recent) history of silly names. BERT, BART, ELMO, RoBERTa, BIGBIRD, PaLM, Megatron etc. Might as well go full nonsense.
True. I will add that it is customary to justify it by demonstrating it is some sort of acronym or contraction.
It's a recursive, selective acronym
C
CH
CHI
CHIN
CHINC
CHINCH
CHINCHI
CHINCHIL
CHINCHILL
==> CHINCHILLA
HINCHILLA
INCHILLA
NCHILLA
CHILLA
HILLA
ILLA
LLA
LA
ARe: DeepMind’s New Language Model, Chinchilla
#20Does this imply we will run out of data to keep up with larger model sizes?
Is there much more data out there than what they’re already using?