Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoGPT

github.com

Post reply on HN
Print view View original

281–290 of 334 posts

Re: NanoGPT

#281

Post by lossolo »

renewiltord wrote: post #273
wongarsu wrote: post #184

Earlier quoted context omitted.

The biggest GPT2 (1.5B params) takes about 10GB VRAM, meaning it runs on a RTX 2080 TI, or the 12GB version of the RTX 3080

What's the largest language model I can run on a 3090 with 24 GiB RAM?

Depends on precision, you can run ~5B model with fp32 precision or ~11B fp16 model max. Int8 is really bad for real world use case so not mentioning it.

But if you are looking to get performance of ChatGPT or GPT-3 then don't waste your time, all GPT-3 like small LLM models (below at least 60B params) are useless for any real world use case, they are just toys.

Re: NanoGPT

#282

Post by visarga »

QuadrupleA wrote: post #217

Doesn't huggingface have dozens of freely available pretrained models like this (including various sized implementations of GPT2) and isn't the source available on most if you wanted to train them yourself? All I see in the comments is praise for the author as a person, so just wondering what's unique about this that's not available elsewhere? 730 upvotes and counting, assuming I'm missing something...

This is a didactic implementation. If you read the HuggingFace repo it is much more abstracted on account they implement many models in the same codebase. It's not fast or big, just easier to read and tweak.

Re: NanoGPT

#283

Post by visarga »

ausbah wrote: post #202
justusthane wrote: post #107

This is a dumb question about language models in general, not necessarily specific to NanoGPT: why is all the focus on training? Can I download and run a pre-trained model locally? Surely the specs required to run a model are much, much lower than those required to train the model?

inference can still be a bottleneck i think since you usually load the whole thing into memory which is 32-64GB+ usually?

Language models range from 1 to 300+ GB when loaded. It depends on how you load them, if you load in int8 you get 4x reduction.

Re: NanoGPT

#284

Post by visarga »

imtringued wrote: post #47
awestroke wrote: post #2

Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to train with incredibly expensive hardware, which means small players have no chance to push the state of the art

As models get bigger less and less neurons get activated by any given input. If you can somehow predict which neurons get activated you can skip the vast majority of the computational load. I have read a paper where they argued that only 0.5% of the neurons are actually active in a 200 million parameter model so you can get a 200x improvement just from that. What this tells you is that there is very little money in o…

> argued that only 0.5% of the neurons are actually active in a 200 million parameter model so you can get a 200x improvement just from that

Yes, but you don't know which 0.5% depending on the input text.

Re: NanoGPT

#285

Post by mryab »

xur17 wrote: post #57
awestroke wrote: post #2

Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to train with incredibly expensive hardware, which means small players have no chance to push the state of the art

Alternatively, are there ways to train on consumer graphics cards, similar to SETI@Home or Folding@Home? I would personally be happy to donate gpu time, as I imagine many others would as well.

There absolutely are! Check out hivemind (https://github.com/learning-at-home/hivemind), a general library for deep learning over the Internet, or Petals (https://petals.ml/), a system that leverages Hivemind and allows you to run BLOOM-176B (or other large language models) that is distributed over many volunteer PCs. You can join it and host some layers of the model by running literally one command on a Linux machine with Docker and a recent enough GPU.

Disclaimer: I work on these projects, both are based on our research over the past three years

Re: NanoGPT

#286

Post by visarga »

antognini wrote: post #222
narrator wrote: post #200

Earlier quoted context omitted.

Jevon's paradox of data and AI. The more efficiently data is used, the more demand their is for data.

Any state of the art model takes about three weeks to train.

More an indication of human patience than task difficulty.

Re: NanoGPT

#287

Post by visarga »

CuriouslyC wrote: post #150
imtringued wrote: post #47

Earlier quoted context omitted.

As models get bigger less and less neurons get activated by any given input. If you can somehow predict which neurons get activated you can skip the vast majority of the computational load. I have read a paper where they argued that only 0.5% of the neurons are actually active in a 200 million parameter model so you can get a 200x improvement just from that. What this tells you is that there is very little money in o…

This is hard a-priori, but fairly easy post-facto. Model distillation isn't a common practice yet, but it has already been demonstrated to be quite effective for specific use cases.

Distillation works but somehow we see very few papers doing it at this scale.

Re: NanoGPT

#288

Post by visarga »

captainmuon wrote: post #23
awestroke wrote: post #2

Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to train with incredibly expensive hardware, which means small players have no chance to push the state of the art

I wonder about this, too. OpenAI's biggest 'moat' is that their model takes so much resources to train, not that their algorithms are particularly secret. One idea I had was to not use one single model to learn all steps of the task, but to break it up. The human brain has dedicated grammar processing parts. It is unclear whether something like a universal grammar exists, but we have at least an innate sense for rhyt…

Check out DeepMind RETRO, it's one year old already, but exactly what you say:

https://www.deepmind.com/publications/improving-language-mod...

Re: NanoGPT

#289

Post by belter »

snerbles wrote: post #243
belter wrote: post #121

Earlier quoted context omitted.

Or $9/hour if you use Spot :-) https://aws.amazon.com/ec2/spot/pricing/

Hopefully your progress gets saved in time when the spot instance inevitably gets terminated in the midst of training.

"Managed Spot Training..."

"...Spot instances can be interrupted, causing jobs to take longer to start or finish. You can configure your managed spot training job to use checkpoints. SageMaker copies checkpoint data from a local path to Amazon S3. When the job is restarted, SageMaker copies the data from Amazon S3 back into the local path. The training job can then resume from the last checkpoint instead of restarting...."

https://docs.aws.amazon.com/sagemaker/latest/dg/model-manage...

Re: NanoGPT

#290

Post by visarga »

ccozan wrote: post #68
captainmuon wrote: post #23

Earlier quoted context omitted.

I wonder about this, too. OpenAI's biggest 'moat' is that their model takes so much resources to train, not that their algorithms are particularly secret. One idea I had was to not use one single model to learn all steps of the task, but to break it up. The human brain has dedicated grammar processing parts. It is unclear whether something like a universal grammar exists, but we have at least an innate sense for rhyt…

GPT and human brain ( at least the language / speech part ) have nothing in common. We, as humans, do not use language in a generative way, is derived from a higher or very low level of abstraction ( intentions, emotions, etc ) and is explictly use for communicating something. Even this text is based on previous knowledge, saved in an abstract way, and while writing this I must follow the synthax of the language or w…

> GPT and human brain have nothing in common

Here we go again. They must have something in common, because for about 90% of the tasks the language model agrees with humans, even on novel tasks.

> We, as humans, do not use language in a generative way

Oh, do you want to say we are only doing classification from a short list of classes and don't generate open ended language? Weird, I speak novel word combinations all the time.

Post reply on HN