Inverted Y axis?! I get marketing probably wanted them on the top right hand corner as “best” but it makes me cringe seeing this.
GigaGPT: GPT-3 sized models in 565 lines of code
11–20 of 70 posts
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#12Earlier quoted context omitted.
Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.
To be honest, HF compat comes at the cost of abysmal performance compared to an inference solution such as vLLM.
New to the space — I have been using HF for both training and inference.
Do you have recommendations for a different framework for training? Accelerate seems fantastic for scaling up once I need to
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#13Looks like hardware vs. software abstraction. Considereing the perspective of an LLM startup: Would you rather write 20k LOC of complex code that would make you be able to more easily switch hardware platforms - or - write 600 LOC of less complex code and be pinned to a single provider?
What I'm most interested in with abstraction is how easy it is to change something that doesn't fit neatly inside of the abstraction framework. It looks like the model is pretty flexible as it's just plain pytorch, I couldn't immediately tell about other aspects of the training - for example they have their own optimizer, what if I want to change something?
There are lots of "just one line of python" type frameworks that are fine if you want to do the one thing in the demo but are more complicated than just writing it yourself if you have to change something.
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#14For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single arc…
This does however feel a bit like that "big data" phenomenon, where most companies deploy ridiculously overcomplicated distributed data clusters, where their actual problems could be handled much more simply, cheaply and efficiently, by a single server with a lots of RAM and a solution somewhere on the spectrum between "bunch of UNIX pipes with standard UNIX text processing tools" and "tuned PostgreSQL" / "tuned in-memory SQLite".
That is: a lot of distributed big data processing tasks don't need to be distributed. Perhaps with beefy enough matrix multiplication chips, a lot of "big ML" tasks won't need to be distributed either.
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#15Earlier quoted context omitted.
Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.
Stay far away from HuggingFace if you can. Battle-hardened if you only do the absolute simplest and boring stuff. Look at the number of open issues and skim through a few source code files and you’ll understand. Like LangChain, they were at the right place at the right time. That doesn’t make them good.
Good to hear that I'm not the only one thinking that. I read a lot of their code for one of my pet projects and I thought that maybe it's me that doesn't get it because I don't write that much python.
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#16I assume the 565 lines of code refer to the # of lines of native code (not counting the # of lines referring to libraries used).
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#17According to https://www.anandtech.com/show/16626/cerebras-unveils-wafer-... the WSE-1 was $2Million, so I expect the WSE-2 is an arm and a leg too.
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#18For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single arc…
This does however feel a bit like that "big data" phenomenon, where most companies deploy ridiculously overcomplicated distributed data clusters, where their actual problems could be handled much more simply, cheaply and efficiently, by a single server with a lots of RAM and a solution somewhere on the spectrum between "bunch of UNIX pipes with standard UNIX text processing tools" and "tuned PostgreSQL" / "tuned in-m…
I'm a big believer in the approach you're laying out too. Bugs are much easier to diagnose, crash reporting is more straightforward, and you don't need augmented services to consolidate everything at the output layer. That said - I've been predicting a shift back to simple architectures for awhile now and they haven't really come to pass. Maybe there's too much pressure or financial incentives for increasing complexity to solve increasing complexity?
At the end of the day I do believe ergonomics are going to win out. I think that's in large part why pytorch won out over tensorflow and jax; it provided the just-in-time computation that would allow people to more easily find bugs & visualize results without having to `compile()` everything down to a static computation graph. Hardware seems like a natural place for that abstraction layer - but maybe the silver bullet will really be on the software side, since we already have too many "low-RAM" equivalent ML devices in the wild. Cheaper to string things together after the fact vs. shipping net new hardware.
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#19Looks like hardware vs. software abstraction. Considereing the perspective of an LLM startup: Would you rather write 20k LOC of complex code that would make you be able to more easily switch hardware platforms - or - write 600 LOC of less complex code and be pinned to a single provider?
I think you're missing a crucial difference here:
> ... write 20k LOC of complex code ...
Vs the all important:
> ... write _and maintain_ 20k LOC of complex code ...
The answer to the latter, for me, is a hard no.
Re: GigaGPT: GPT-3 sized models in 565 lines of code
#20For those hearing about Cerebras for the first time, they make a chipset that's similar to a GPU in matrix multiplication speed but way bigger (a whole wafer) so it can fit more transistors and memory onto one chip. They achieve this small LOC count because they don't need to shard across multiple devices / backprop consolidate on a central CPU / etc. These tricks are usually what blows up a project from a single arc…
This does however feel a bit like that "big data" phenomenon, where most companies deploy ridiculously overcomplicated distributed data clusters, where their actual problems could be handled much more simply, cheaply and efficiently, by a single server with a lots of RAM and a solution somewhere on the spectrum between "bunch of UNIX pipes with standard UNIX text processing tools" and "tuned PostgreSQL" / "tuned in-m…
You can hide a whole lot of essential complexity in a hardware layer.
However, the very next question a researcher will ask once a model fits on one device is “can I make it twice as fast/big if I use two?”