Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GigaGPT: GPT-3 sized models in 565 lines of code

cerebras.net

Post reply on HN
Print view View original

51–60 of 70 posts

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#51

Post by choppaface »

jpetrucc wrote: post #48
icyfox wrote: post #7

For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single arc…

Are these chipsets anywhere in the price range that would make them feasible for consumer/pro-sumer? I wasn't able to find anything related to pricing without contacting sales. With the numbers being thrown about, this would appear to be "enterprise-only"

No, three years ago the CS1 was already $1-2m minimum per chip https://www.anandtech.com/show/15838/cerebras-wafer-scale-en...

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#52

Post by choppaface »

WhitneyLand wrote: post #50
icyfox wrote: post #7

For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single arc…

I didn’t notice performance mentioned anywhere in the article and I think that’s not a good sign at all.

For transformers, especially multi-device training pipelines, yes the codebase can normally be 100ksloc and require a team to do at industry scale. See e.g. Hugginface Transformers, or the Megatron impl they cite.

Cerebras is trying to show how easy it is to on-board single ICs and demo their pytorch integration.

But yeah, where's the wallclock time comparison?! Surely they did one during development, and surely the Sales team knows (or they do once the article was published), yet not even a hint of what their throughput is like. For Cerebras to be this far and not be plastering benchmarks everywhere is a bad sign. Maybe they're going to just die off like Graphcore.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#53

Post by esafak »

happytiger wrote: post #37
icyfox wrote: post #18

Earlier quoted context omitted.

I'm a big believer in the approach you're laying out too. Bugs are much easier to diagnose, crash reporting is more straightforward, and you don't need augmented services to consolidate everything at the output layer. That said - I've been predicting a shift back to simple architectures for awhile now and they haven't really come to pass. Maybe there's too much pressure or financial incentives for increasing complexi…

Which architecture more closely resembles the human brain?

Spiking neural networks, I think.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#55

Post by rockinghigh »

natch wrote: post #46

Ignorant question: Why are we interested in training models much smaller than GPT-4? For academic reasons? I understand training in specific domains but isn’t that covered by fine tuning, with much less compute?

Cheaper and faster inference. Cheaper fine tuning.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#56

Post by _boffin_ »

natch wrote: post #46

Ignorant question: Why are we interested in training models much smaller than GPT-4? For academic reasons? I understand training in specific domains but isn’t that covered by fine tuning, with much less compute?

They’re doing marketing to sell more shovels as they’re a shovel selling company.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#58

Post by spacecadet »

TeMPOraL wrote: post #14
icyfox wrote: post #7

For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single arc…

This does however feel a bit like that "big data" phenomenon, where most companies deploy ridiculously overcomplicated distributed data clusters, where their actual problems could be handled much more simply, cheaply and efficiently, by a single server with a lots of RAM and a solution somewhere on the spectrum between "bunch of UNIX pipes with standard UNIX text processing tools" and "tuned PostgreSQL" / "tuned in-m…

Agreed and thank you for also posting. I haven't incurred any cloud costs and my home server upgrades have been done for cheap, if you really look. I laughed because my "pipe" is as you put it, "Unix pipes with standard text processing". Hey, it works.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#59

Post by Paul-Craft »

bubblethink wrote: post #47
Paul-Craft wrote: post #44

Earlier quoted context omitted.

You could just... read the code: https://github.com/Cerebras/gigaGPT

I think the point is that LOC is not a terribly useful metric in that everything is 1 LOC at the highest level of abstraction. The business proposition here is that you don't need to write the LOCs for the underlying layers, they do it. The pitch here is that it's not as straightforward for large GPU clusters.

Sure, I get that. I've definitely seen demos of "Do X in Y LoC" that do X but offload all of the hard work of Y to some libray. This is not that. This is intended to be a demo that shows you what you can do with one Cerebras module. And, the result is that, by writing 565 LoC yourself, you can train and run an LLM the size of GPT-3.

In that sense, 565 LoC is a perfectly fair number. It doesn't count PyTorch, numpy, the Python interpreter, or any of the library modules that are imported, but I don't think anyone was touting it as anything more; for instance, Mo Gawdat has said that GPT-4 is probably ~4500 LoC. And, yes, that certainly involves much more infrastructure, and doing that dance of going from GPU to CPU to a completely other node, etc.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#60

Post by [deleted] »

choppaface wrote: post #52
WhitneyLand wrote: post #50

Earlier quoted context omitted.

I didn’t notice performance mentioned anywhere in the article and I think that’s not a good sign at all.

For transformers, especially multi-device training pipelines, yes the codebase can normally be 100ksloc and require a team to do at industry scale. See e.g. Hugginface Transformers, or the Megatron impl they cite. Cerebras is trying to show how easy it is to on-board single ICs and demo their pytorch integration. But yeah, where's the wallclock time comparison?! Surely they did one during development, and surely the…

[deleted]
Post reply on HN