GigaGPT: GPT-3 sized models in 565 lines of code
cerebras.net
GigaGPT: GPT-3 sized models in 565 lines of code
1–10 of 70 posts
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#2[deleted]
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#3Looks like hardware vs. software abstraction. Considereing the perspective of an LLM startup: Would you rather write 20k LOC of complex code that would make you be able to more easily switch hardware platforms - or - write 600 LOC of less complex code and be pinned to a single provider?
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#4Looks like hardware vs. software abstraction. Considereing the perspective of an LLM startup: Would you rather write 20k LOC of complex code that would make you be able to more easily switch hardware platforms - or - write 600 LOC of less complex code and be pinned to a single provider?
Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#5[deleted]
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#6Looks like hardware vs. software abstraction. Considereing the perspective of an LLM startup: Would you rather write 20k LOC of complex code that would make you be able to more easily switch hardware platforms - or - write 600 LOC of less complex code and be pinned to a single provider?
Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.
To be honest, HF compat comes at the cost of abysmal performance compared to an inference solution such as vLLM.
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#7For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single architecture proof of concept to a robust training pipeline that can handle the billions of parameters on modern models. This is more akin to training a whole model on a single GPU because... it kind of is.
Even with a wafer scale chipset this approach has limits. You eventually will still need to shard to fit more parameters / use different training modalities / etc. I'd look at this more as a proof of concept for the ergonomics of what LLM training can look like when you have access to a much larger compute primitive versus a new state of the art in feature-equivalent clean code.
Disclaimer: I'm a small investor in Cerebras.
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#8Looks like hardware vs. software abstraction. Considereing the perspective of an LLM startup: Would you rather write 20k LOC of complex code that would make you be able to more easily switch hardware platforms - or - write 600 LOC of less complex code and be pinned to a single provider?
Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.
Stay far away from HuggingFace if you can. Battle-hardened if you only do the absolute simplest and boring stuff. Look at the number of open issues and skim through a few source code files and you’ll understand.
Like LangChain, they were at the right place at the right time. That doesn’t make them good.
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#9I would’ve been interested to learn how much it costs to train these models using their platform. Like, a 70b model - are we talking millions of dollars here?
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#10Looks like hardware vs. software abstraction. Considereing the perspective of an LLM startup: Would you rather write 20k LOC of complex code that would make you be able to more easily switch hardware platforms - or - write 600 LOC of less complex code and be pinned to a single provider?
Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.
We are talking about very big models which training requires an enormous amount of hardware. Not sure how scalable HuggingFace transformer for training such models is.