Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DeepMind’s New Language Model, Chinchilla

marktechpost.com

Post reply on HN
Print view View original

21–30 of 145 posts

Re: DeepMind’s New Language Model, Chinchilla

#21

Post by omarhaneef »

yeetsfromhellL2 wrote: post #19
omarhaneef wrote: post #14

Earlier quoted context omitted.

True. I will add that it is customary to justify it by demonstrating it is some sort of acronym or contraction.

It's a recursive, selective acronym C CH CHI CHIN CHINC CHINCH CHINCHI CHINCHIL CHINCHILL ==> CHINCHILLA HINCHILLA INCHILLA NCHILLA CHILLA HILLA ILLA LLA LA A

I know what recursive means, I know what selective means, I know what an acronym is, and I think I see the pattern in that picture, but when I put it all together I am lost.

Alternatively, is this a joke and the "recursive, selective acronym" can be used to justify any word?

Re: DeepMind’s New Language Model, Chinchilla

#23

Post by sirk390 »

guipsp wrote: post #16
g051051 wrote: post #15

Is there a good reference as to what a "parameter" is in this context? I've looked a few times, but the explanations don't make any sense to me.

You can think of a parameter as a number you can tweak while training. This network has 70B such numbers.

And if every parameter is one byte, the minimum, it will take at least 70gb to save or share this model. So it's still way to big to package directly in a app.

Re: DeepMind’s New Language Model, Chinchilla

#25

Post by eigenvalue »

ShamelessC wrote: post #5

Seems the link is down. Found a decent synopsis/discussion on lesswrong. https://www.lesswrong.com/posts/midXmMb2Xg37F2Kgn/new-scalin... > On March 29th, DeepMind published a paper, "Training Compute-Optimal Large Language Models", that shows that essentially everyone -- OpenAI, DeepMind, Microsoft, etc. -- has been training large language models with a deeply suboptimal use of compute. > Following the new scaling la…

According to the LessWrong post, the smaller model trained on more data performs better on most of the tasks, but it’s worse on “college level math” questions. I wonder why that is. Is it because the extra capacity of the larger model was used to basically memorize theorems? Or is it because the extra “brain power” let it model the math better? Oddly, one of the tasks that the smaller most outperformed the larger model on is “high school level math”! Very counterintuitive, and I am curious if there are any big takeaways lurking in that disparity.

Re: DeepMind’s New Language Model, Chinchilla

#26

Post by isaacfrond »

They trained over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens while staying under a given compute budget. The results are modelled, and they pick the best one. Turns out the having a bit fewer tokens improves performance.

Re: DeepMind’s New Language Model, Chinchilla

#27

Post by cshimmin »

g051051 wrote: post #15

Is there a good reference as to what a "parameter" is in this context? I've looked a few times, but the explanations don't make any sense to me.

It's a degree of freedom of the learnable model. For example in a "vanilla" neural network layer (MLP), which maps from M to N feature dimensions will contain an MxN matrix of learnable parameters that model the connections between the M inputs to the N outputs. Every time the model is updated during backpropagation, the loss gradient which has to be computed has the same dimensionality as the number of parameters. Also, generally more parameters means more operations in the forward pass. Therefore, a model with more parameters in general will require more FLOPs per iteration of training. The main point of this paper is that you can actually do better by training a smaller model for longer, rather than a bigger model for less time, assuming you have a fixed FLOP budget.

Re: DeepMind’s New Language Model, Chinchilla

#29

Post by veonik »

omarhaneef wrote: post #21
yeetsfromhellL2 wrote: post #19

Earlier quoted context omitted.

It's a recursive, selective acronym C CH CHI CHIN CHINC CHINCH CHINCHI CHINCHIL CHINCHILL ==> CHINCHILLA HINCHILLA INCHILLA NCHILLA CHILLA HILLA ILLA LLA LA A

I know what recursive means, I know what selective means, I know what an acronym is, and I think I see the pattern in that picture, but when I put it all together I am lost. Alternatively, is this a joke and the "recursive, selective acronym" can be used to justify any word?

               A
              AR
             ARB
            ARBI
           ARBIT
          ARBITR
         ARBITRA
        ARBITRAR
  ==>  ARBITRARY
       RBITRARY
       BITRARY
       ITRARY
       TRARY
       RARY
       ARY
       RY
       Y
Yup, seems it works for any word.

Re: DeepMind’s New Language Model, Chinchilla

#30

Post by MisterTea »

binarymax wrote: post #9
galcerte wrote: post #3

I have to ask, why call it that? I had a chuckle once I saw the name.

Large language models have a (recent) history of silly names. BERT, BART, ELMO, RoBERTa, BIGBIRD, PaLM, Megatron etc. Might as well go full nonsense.

My theory is since no one reads literature anymore, timeless, interesting and unique names from history and other cultures are lost to a deluge of soon to be forgotten gag, pop-culture and meme names. Perhaps this is why we have Chinchilla and not Oberon.
Post reply on HN