Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GigaGPT: GPT-3 sized models in 565 lines of code

cerebras.net

Post reply on HN
Print view View original

41–50 of 70 posts

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#41

Post by mabrowning »

andy99 wrote: post #13
kkzz99 wrote: post #3

Looks like hardware vs. software abstraction. Considereing the perspective of an LLM startup: Would you rather write 20k LOC of complex code that would make you be able to more easily switch hardware platforms - or - write 600 LOC of less complex code and be pinned to a single provider?

What I'm most interested in with abstraction is how easy it is to change something that doesn't fit neatly inside of the abstraction framework. It looks like the model is pretty flexible as it's just plain pytorch, I couldn't immediately tell about other aspects of the training - for example they have their own optimizer, what if I want to change something? There are lots of "just one line of python" type frameworks…

We do have reference optimizers implemented for use on our system and available in the `cerebras_pytorch` package, but this isn't because those are the only ones supported; instead, no vanilla pytorch optimizer is currently `torch.compile()` compatible. The main difference is that we pre-initialize the optimizer state instead of doing it lazily in the first `step()`

You can read more here https://docs.cerebras.net/en/latest/wsc/tutorials/custom-opt...

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#42

Post by blobbers »

I'm curious if these low-code models matter. I understand that small codebases can be cached effectively speeding up computations, but isn't it the data load the bottleneck in training?

Furthermore, how important is the breadth of data in the dataset to getting the desired results? I was under the impression that the main reason these LLM work is based on massive data sets.

As such, is there data-breadth metrics to validate whether training on a given dataset is even worthwhile? (ie: avoid sunk cost on a dataset that will yield a poorly performing LLM)

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#43

Post by TeMPOraL »

kristjansson wrote: post #34
TeMPOraL wrote: post #23

Earlier quoted context omitted.

I'd think that such researcher would've already heard of adages like "you can't get nine women to give birth in one month", or "where there's six cooks, there's nothing to eat". Or more directly, perhaps one should ask such researcher, "if your team was to double in head count, would you do this project twice as fast?".

8 GPUs do a pretty bang-up job of doing 8 months of compute in a month :). I think the broader point is that the last x0 years of ML research show that more compute is better, both for iteration speed and for resulting performance. Distribution is just the natural outgrowth of that imperative once it reaches the limit of a single device/node. If Cerebras can address models at today's scale on one device, the immediat…

Fair enough :).

I still think work on improving single-core/device performance is worthwhile, as distribution will always strictly not-better, and almost always strictly worse, due to coordination costs reducing efficiency. If two Cerberas can be glued together and achieve roughly 2x of their performance, it's still going to be more efficient than achieving equivalent performance from many more regular GPUs. Getting the hardware fast enough so that you need just one device for your problem - that's a special case that will yield extra win.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#45

Post by nerpderp82 »

jwan584 wrote: post #39
filterfiber wrote: post #35

I don't understand why they're comparing the parameter sizes to lines of code. AFAIK you can just increase the layer parameters of a 1B model to whatever you want? Like, the difference between a 1B and 175B model can be just changing a few numbers, and not adding any LOC at all? LOC has never been a limitation for large models, it's been the compute+training data required. Most of the LOC is spent on optimization, an…

when you go from 1B to 175B, the model no longer fits in memory. so in practice you have to re-factor the model using tensor/pipeline parallelism. that's why it goes from 600 to 20K LOC.

It doesn't look like Cerebras mentioned the most important part, by trading model complexity due to using a vastly more capable system, they could could refactor that 600 line model effortlessly and rerun.

They can watch different layers train and find out how to optimize training or quantization, etc.

It feels like they kinda missed the forest for the trees here. The article should have focused on model architecture optimization due to the small LoC and the system having ridiculous training capacity.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#47

Post by bubblethink »

Paul-Craft wrote: post #44
I_am_tiberius wrote: post #16

I assume the 565 lines of code refer to the # of lines of native code (not counting the # of lines referring to libraries used).

You could just... read the code: https://github.com/Cerebras/gigaGPT

I think the point is that LOC is not a terribly useful metric in that everything is 1 LOC at the highest level of abstraction. The business proposition here is that you don't need to write the LOCs for the underlying layers, they do it. The pitch here is that it's not as straightforward for large GPU clusters.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#48

Post by jpetrucc »

icyfox wrote: post #7

For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single arc…

Are these chipsets anywhere in the price range that would make them feasible for consumer/pro-sumer? I wasn't able to find anything related to pricing without contacting sales. With the numbers being thrown about, this would appear to be "enterprise-only"

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#49

Post by whimsicalism »

jwan584 wrote: post #38

Everyone knows Cerebras by their wafer scale chips. The less understood part is the 12TB of external memory. That's the real reason why large models fit by default and you don't have to chop it up in software ala megatron/deepspeed.

imo the benefits to chopping it up will always remain

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#50

Post by WhitneyLand »

icyfox wrote: post #7

For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single arc…

I didn’t notice performance mentioned anywhere in the article and I think that’s not a good sign at all.
Post reply on HN