21–30 of 231 posts
Slightly off-topic: I remember seeing news about the enormous chip Cerebras was/is selling (pdf https://f.hubspotusercontent30.net/hubfs/8968533/WSE-2%20Dat... ). Has there been any indication that the LLMs released in the last few months use exotic hardware like this, or is it all "standard" hardware?
That was the largest that had inference enabled - I'd really like to try this one: https://huggingface.co/cerebras/Cerebras-GPT-13B
> Trained using the Chinchilla formula, these models provide the highest accuracy for a given compute budget. I'm confused as to why 111 million parameter models are trained with the Chinchilla formula. Why not scale up the training data? If you're training smaller models, surely optimizing performance is better than optimizing total compute. Seems like a silly misunderstanding of the Chinchilla paper, but I'm sure I…
As an example the BERT/RoBERTa family were trained for much longer than Chinchilla, you do get diminishing returns though.
There is a point of overtraining where downstream performance is impacted but that’s pretty high.
I think part of the answer to this is also that xxx million parameter decoder-only models don’t seem to be that useful so it may not be worthwhile to optimize them for performance?
FYI: Cerebras's nodes are very different than your typical Nvidia training nodes: https://www.anandtech.com/show/16626/cerebras-unveils-wafer-... Each individual "chip" has 40GB of SRAM vs ~76MB for the Nvidia H100, and networked pools of external RAM, SSDs and such. Thats why the training architecture is so different.
There's a comparison picture there of one of their chips alongside a regular GPU chip. Effectively they use up the entire wafer.
[1] https://www.hpcwire.com/2021/09/16/cerebras-wafer-scale-engi....
EDIT: Ok, looks like I've missed the hugging face repo. The language they use is a bit confusing.
The cat is out of the bag, the genie is out of the bottle, the confetti has left the cannon[0].
It's tempting to see a world dominated by Google Bard, ChatGPT, Bing Search, etc. And no doubt, they will be huge players, with services that are far more powerful than anything that can be run on the edge.
But. BUT. The things that we can do on the edge are incredible now. Just imagine a year from now, or two. These earth-shattering models, which seem to be upending a whole industry, will soon have equivalents that run on the edge. Without services spying on your data. Without censorship on what the model can/cannot say. Because it's all local.
When was the last time this happened? There will be players who publish weights for models that are free to use. The moment that torrent magnet link is published, it's out in the wild. And smart people will package them as "one click installers" for people who aren't tech-savvy. This is already happening.
So every time you're amazed by something chat-gpt4 says, remember that soon this will be in your pocket.
[0] the "confetti" idiom brought to you by chat-gpt4.
> Trained using the Chinchilla formula, these models provide the highest accuracy for a given compute budget. I'm confused as to why 111 million parameter models are trained with the Chinchilla formula. Why not scale up the training data? If you're training smaller models, surely optimizing performance is better than optimizing total compute. Seems like a silly misunderstanding of the Chinchilla paper, but I'm sure I…
True. There was a good blog post published about this a few weeks ago: https://finbarr.ca/llms-not-trained-enough/ Money quote for those who don't want to read the whole thing: ''' When people talk about training a Chinchilla-optimal model, this is what they mean: training a model that matches their estimates for optimality. They estimated the optimal model size for a given compute budget, and the optimal number of t…
I might be missing something but it looks to me that actually running this "open" model requires special hardware only accessible with a cloud subscription with 60 000 USD / week minimum spend[1]. Can anyone confirm if you can run it on your own hardware? If software is open but hardware is locked I don't see the point. [1] https://www.hpcwire.com/2021/09/16/cerebras-wafer-scale-engi... . EDIT: Ok, looks like I've mi…
I might be missing something but it looks to me that actually running this "open" model requires special hardware only accessible with a cloud subscription with 60 000 USD / week minimum spend[1]. Can anyone confirm if you can run it on your own hardware? If software is open but hardware is locked I don't see the point. [1] https://www.hpcwire.com/2021/09/16/cerebras-wafer-scale-engi... . EDIT: Ok, looks like I've mi…
I wonder what led to such a gap between llama 7b and Cerebras 13b. I hope they discuss it in the paper.
Edit: The huggingface page has 0-shot benchmarks which you can compare against the llama paper