Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GigaGPT: GPT-3 sized models in 565 lines of code

cerebras.net

Post reply on HN
Print view View original

21–30 of 70 posts

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#21

Post by tbalsam »

minimaxir wrote: post #4
kkzz99 wrote: post #3

Looks like hardware vs. software abstraction. Considereing the perspective of an LLM startup: Would you rather write 20k LOC of complex code that would make you be able to more easily switch hardware platforms - or - write 600 LOC of less complex code and be pinned to a single provider?

Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.

HF is okay for off the shelf prototypes and very quick, multi-hour hacks, and I greatly appreciate their contributions to the community, but unfortunately in my (and sadly, many other people's experiences), their code is a continuum of devilish nightmares for anything beyond that.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#23

Post by TeMPOraL »

kristjansson wrote: post #20
TeMPOraL wrote: post #14

Earlier quoted context omitted.

This does however feel a bit like that "big data" phenomenon, where most companies deploy ridiculously overcomplicated distributed data clusters, where their actual problems could be handled much more simply, cheaply and efficiently, by a single server with a lots of RAM and a solution somewhere on the spectrum between "bunch of UNIX pipes with standard UNIX text processing tools" and "tuned PostgreSQL" / "tuned in-m…

You can hide a whole lot of essential complexity in a hardware layer. However, the very next question a researcher will ask once a model fits on one device is “can I make it twice as fast/big if I use two?”

I'd think that such researcher would've already heard of adages like "you can't get nine women to give birth in one month", or "where there's six cooks, there's nothing to eat".

Or more directly, perhaps one should ask such researcher, "if your team was to double in head count, would you do this project twice as fast?".

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#24

Post by 101008 »

Would it be ever possible to run a GPT-{n}, n>3, similar model in a home computer wihtout GPU? I have a "good" laptop with 32GB, good processor, but no GPU (I was never interested in gaming, crypto or ML), but I found GPT very useful and I'd prefer to run a local version instead of keep feeding OpenAI.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#25

Post by minimaxir »

jackblemming wrote: post #8
minimaxir wrote: post #4

Earlier quoted context omitted.

Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.

Stay far away from HuggingFace if you can. Battle-hardened if you only do the absolute simplest and boring stuff. Look at the number of open issues and skim through a few source code files and you’ll understand. Like LangChain, they were at the right place at the right time. That doesn’t make them good.

In the context of this blog post (reimplementing nanoGPT as-is without shenanigans, although training is a separate issue), transformers is straightforward enough. I do agree that going beyond the documentation demos does make things more complicated, but IMO still less so than other implementations speaking from experience. AI is still sometimes necessarily complex.

It's certainly an order of magnitude easier to use something like transformers or diffusers than the original implementations provided by the original model trainers, and has a few good optimizations out of the box.

That's different from LangChain which is complex for the sake of being complex.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#26

Post by minimaxir »

kkzz99 wrote: post #10
minimaxir wrote: post #4

Earlier quoted context omitted.

Or use a more battle-harded abstraction like Hugging Face transformers and get both a) minimal LOC in the end-application and b) portability.

We are talking about very big models which training requires an enormous amount of hardware. Not sure how scalable HuggingFace transformer for training such models is.

transformers outsources those types of tasks to accelerate, also developed by Hugging Face: https://huggingface.co/docs/accelerate/index

I can't speak to the efficacy of it at large scale, though.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#27

Post by andy99 »

TeMPOraL wrote: post #23
kristjansson wrote: post #20

Earlier quoted context omitted.

You can hide a whole lot of essential complexity in a hardware layer. However, the very next question a researcher will ask once a model fits on one device is “can I make it twice as fast/big if I use two?”

I'd think that such researcher would've already heard of adages like "you can't get nine women to give birth in one month", or "where there's six cooks, there's nothing to eat". Or more directly, perhaps one should ask such researcher, "if your team was to double in head count, would you do this project twice as fast?".

> Or more directly, perhaps one should ask such researcher, "if your team was to double in head count, would you do this project twice as fast?".

If everybody's job was just to do dot products all day I'd hope the answer would be yes.

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#28

Post by whynotmaybe »

101008 wrote: post #24

Would it be ever possible to run a GPT-{n}, n>3, similar model in a home computer wihtout GPU? I have a "good" laptop with 32GB, good processor, but no GPU (I was never interested in gaming, crypto or ML), but I found GPT very useful and I'd prefer to run a local version instead of keep feeding OpenAI.

Have you checked Gpt4all or Faraday.dev ?

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#29

Post by [deleted] »

101008 wrote: post #24

Would it be ever possible to run a GPT-{n}, n>3, similar model in a home computer wihtout GPU? I have a "good" laptop with 32GB, good processor, but no GPU (I was never interested in gaming, crypto or ML), but I found GPT very useful and I'd prefer to run a local version instead of keep feeding OpenAI.

[deleted]

Re: GigaGPT: GPT-3 sized models in 565 lines of code

#30

Post by mejutoco »

andy99 wrote: post #27
TeMPOraL wrote: post #23

Earlier quoted context omitted.

I'd think that such researcher would've already heard of adages like "you can't get nine women to give birth in one month", or "where there's six cooks, there's nothing to eat". Or more directly, perhaps one should ask such researcher, "if your team was to double in head count, would you do this project twice as fast?".

> Or more directly, perhaps one should ask such researcher, "if your team was to double in head count, would you do this project twice as fast?". If everybody's job was just to do dot products all day I'd hope the answer would be yes.

Which dot products did you do, I'll do the next one. Oh, that was John's, but he is away on vacation today. Let me take care of John's and tomorrow we have a quick meeting to see which matrix he takes next. Sounds a lot like a bus :)
Post reply on HN