Cerebras-GPT: Open Compute-Optimal Language Models Trained on Cerebras Cluster • Hacker News Board

Re: Cerebras-GPT: Open Compute-Optimal Language Models Trained on Cerebras Cluster

#11

Post by groodt » Sat, Apr 08, 2023, 7:34 AM UTC

cs-fan-101 wrote: post #2

Recently, we announced in this post ( https://news.ycombinator.com/item?id=35343763#35345980 ) the release of Cerebras-GPT — a family of open-source GPT models trained on the Pile dataset using the Chinchilla formula. Today, we are excited to announce the availability of the Cerebras-GPT research paper on arXiv.

Thanks for publishing this. I quickly skimmed the paper, I saw the impressive linear scaling as you scaled to 16 nodes. How long did it take to train the various models in wall clock time?

Re: Cerebras-GPT: Open Compute-Optimal Language Models Trained on Cerebras Cluster

#12

Post by vorticalbox » Sat, Apr 08, 2023, 8:29 AM UTC

You can download these models on huggingface[0]

[0] https://huggingface.co/cerebras

Re: Cerebras-GPT: Open Compute-Optimal Language Models Trained on Cerebras Cluster

#13

Post by ersiees » Sat, Apr 08, 2023, 10:11 AM UTC

Very interesting that someone finally tries out muP in the real world. Do I understand the usage correctly:

MuP is only used to get around choosing an lr for each size? Here I wonder how it compares to standard heuristics like the one in the OG scaling laws paper by OAI and tricks like back winding a few steps after loss explosion.

For some reason muP was not trusted with the largest trainings? Why is that?