71–80 of 145 posts
Earlier quoted context omitted.
I think there remains an immense amount of such suboptimality still hanging from the tree, so to speak. For example, our recent paper "Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer"[1] shows that even learning rate and initialization used by existing models are deeply wrong . By just picking them correctly (which involves some really beautiful mathematics), we can effectively d…
What do you think about the concept of "critical batch size"? https://openai.com/blog/science-of-ai/
But as far as I know, the more precise predictions of optimal batch size aren't used much, probably because it's expensive to measure accurately, or because the predictive equation isn't accurate enough to begin with. I wonder if we can "transfer" the optimal batch size from a smaller setting (smaller model or data) to the full setting, like in our paper. This would make it much more practical.
Earlier quoted context omitted.
From the paper, they are using bfloat16, so I guess two bytes. But distributing and "packaging into an app" are not at all of practical interest for these kinds of models. You (a consumer) would interact via some API service, with the model running on a hardware-accelerated compute cloud. In any case, during training (where the model is run in possibly large batches), and even during inference, the size of the parame…
> even during inference, the size of the parameters is completely dwarfed by the intermediate tensor representations What makes you say this?
Something similar applies for RNNs (same weights applied at each element of a sequence), GNNs and transformers (same weights applied at each pair of data).
Earlier quoted context omitted.
Gwern responded to a similar question in the comments section. (parent) > the fact that data and compute need to scale proportionally seems… like a big point in favor of NNs as memorizers/interpolators. (gwern) > Surely it's the opposite? The more bang you get out of each parameter, the less it looks like 'just' (whatever that means) memorization/interpolation. When you needed to increase parameters a lot, disproport…
This isn’t addressing their question. And Gwern’s goal here is to (incorrectly) try to get rid of the idea that models are just memorizing and interpolating, when in fact memorization and interpolation is what we all do, including models. He’s just bothered by the idea that people think of models as less than magic. On the other hand, https://twitter.com/model_mechanic/status/151297688118364569... is admittedly prett…
From your comment a reader will understand that you think they're just memorizing and interpolating and that you disagree with gwern on this point, but you've given your reader nothing that argues in favor of your position
Why should someone believe that models are just memorizing and interpolating?
Earlier quoted context omitted.
Aren’t most of the models at the top not suitable for text generation? That’s what makes gpt different from Bert
What are the models at the top used for? Excuse my ignorance.
Earlier quoted context omitted.
Is there any source which explains what billion of parameters actually are? In my mind a parameter is: language, dialect, perhaps context parameters (food, dinner, lunch, travel) and if we than talk about language and audio perhaps sound waves, gender. Or are context parameters which gives you insight? Like a billion of parameters are literally something like travel=false, travel-europe=true people speaking=e, age, h…
A parameter is a scalar value, most of which are in the attention matrices and feedforward matrices, you also hear these called “weights”. Any intro to DL course will cover these in detail. I recommend started with Andrew Ng’s Coursera class on Intro to Machine Learning, although there may be better ones out there now.
I see tx
Off-topic to Chinchilla, but relevant to the source site: MarkTechPost consistently borderline plagiarizes articles and shares them on their website as "paper summaries". They copy-paste from the source material and change some of the wording around as to appear original. My work, as well as other work from Berkeley AI Research, has been posted in this manner on their site. This seems highly unethical, and I'm surpri…
Cloudflare will forward it to their host, I believe, who will then ask that they remove the infringing material, or provide a counter claim.
Earlier quoted context omitted.
What are the models at the top used for? Excuse my ignorance.
Mostly mask fill, but Transformers can be fine tuned to downstream tasks relatively easily (T5 was built for translation but is used for autocomplete in many cases)
Earlier quoted context omitted.
Probably right. Most people dump on these language models for this reason but it would be absurd for a HS student to have to re-derive the quadratic equation every time they worked on an Algebra problem so naturally you memorize it. Why should it be any different for a language model?
I never memorized the quadratic formula, and I did OK.
Earlier quoted context omitted.
the biggest problem first of all might be the memory requirements given so many parameters. It couldn't be as cheap as a high end computer in the foreseeable future.
There is probably a space-time trade off that needs to be explored in this space. It might be possible to preload the some of the most likely tokens to be selected next into the cache and/or RAM. These are glorified auto-complete algorithms that are poorly understood, as DeepMind's optimizations appear to show. For the English language, it is probable that there are only so many possible grammatically correct selecti…
Can you say that about any other task in ML? When Inceptionv3 came out I was able to run the model pretty comfortable on a 1060. Even pix2pix and most GANs fit comfortably in commercial compute, and the top of the line massive models can still run inference on a 3090. It’s so unbelievably ironic that one of the major points Transformers aimed to solve when introduced was the compute inefficiency of recurrent networks, and it’s devolved into “how many TPUs can daddy afford” instead.