Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoGPT

github.com

Post reply on HN
Print view View original

271–280 of 334 posts

Re: NanoGPT

#271

Post by amelius »

mk_stjames wrote: post #262
amelius wrote: post #111

Earlier quoted context omitted.

It would be great if a tradeoff could be made, though. For example, train at 1/10th the speed for 1/10th of the cost. This could correspond to taking public transport in your analogy, and would bring this within reach of most students.

The problem with that is currently, the available memory scales with the class of GPU.... and very large language models need 160-320GB of VRAM. So, there sadly isn't anything out there that you can load up a model this large on except a rack of 8x+ A40s/A100s. I know there are memory channel bandwidth limits and whatnot but I really wish there was a card out there with a 3090 sized die but with 96GB of VRAM solely t…

I guess this would only become a reality if games started requiring these cards.

Re: NanoGPT

#272

Post by ofcourseyoudo »

pavlov wrote: post #89
arturventura wrote: post #76

This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.

I don't know if that's a blocker. Ordinary people commonly rent a $40k machine for 38 hours from companies like Avis and Hertz. If training a large model now costs the same as driving to visit grandma, that seems like a pretty good deal.

Similarly maybe we should only let people rent a NanoGPT box if they are over 25 and they have to get collision insurance.

Re: NanoGPT

#273

Post by renewiltord »

wongarsu wrote: post #184
bilsbie wrote: post #168

Earlier quoted context omitted.

What’s required to run the model?

The biggest GPT2 (1.5B params) takes about 10GB VRAM, meaning it runs on a RTX 2080 TI, or the 12GB version of the RTX 3080

What's the largest language model I can run on a 3090 with 24 GiB RAM?

Re: NanoGPT

#274

Post by TheAlchemist »

karpathy wrote: post #225

Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.

Thank you for your amazing work. Between cs231n and your recent videos, I've learned a ton - and you have a gift to explain things in such an easy and straightforward way, that I'm always feeling like an idiot (in a positive way) for not having grasped the concept before.

Re: NanoGPT

#275

Post by jwithington »

marviel wrote: post #263
randoglando wrote: post #261

Earlier quoted context omitted.

How does it compare to fast.ai? As a engineer looking to learn, what should I start with?

Both are good for different things. Fast.AI is great, but it takes the top down, vs the bottom up, approach. It takes you from a production-level black box that you don't understand, down to the details. The benefit there is you get good high-level intuition of how it behaves at the "let me use this technology for a job" level. Separately, the fast.ai library is also highly recommendable -- it comes with some state-o…

Jeremy @ Fast.ai says he takes this pedagogical approach because it's "proven" to be the best way to learn. He's probably right, but I do find it confusing at times because in the beginning you're just hitting ctrl + enter on a IPYNB haha.

Maybe Karpathy's approach will speak to me more--thanks for the recommendation!

Re: NanoGPT

#276

Post by dsabanin »

karpathy wrote: post #225

Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.

Thank you for your great work!

Re: NanoGPT

#277

Post by taylorius »

Wow, this is great. I can't wait for the video lecture, transformers are an aspect of modern machine learning that I'm not completely clear on. Andrej's lectures are brilliant - super detailed, and really answer the detailed questions I always have. Great stuff!

Re: NanoGPT

#278

Post by jwithington »

What would I google to figure out how to productionize the output of this?

This repo trains a model--how would I prompt it and print the generated output?

Re: NanoGPT

#279

Post by eternalban »

karpathy wrote: post #225

Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.

Thank you for sharing your knowledge. Anything that can be done to democratize machine learning is an invaluable social service. Hats off to you.

Re: NanoGPT

#280

Post by buildbot »

mk_stjames wrote: post #262
amelius wrote: post #111

Earlier quoted context omitted.

It would be great if a tradeoff could be made, though. For example, train at 1/10th the speed for 1/10th of the cost. This could correspond to taking public transport in your analogy, and would bring this within reach of most students.

The problem with that is currently, the available memory scales with the class of GPU.... and very large language models need 160-320GB of VRAM. So, there sadly isn't anything out there that you can load up a model this large on except a rack of 8x+ A40s/A100s. I know there are memory channel bandwidth limits and whatnot but I really wish there was a card out there with a 3090 sized die but with 96GB of VRAM solely t…

Technically this is not true- there are a lot of techniques to shard models and store activation between layers or even smaller subcomponents of the network. For example, you can split the 175B parameter bloom model into separate layers, load up a layer, read the prev. layers input from disk, and save the output to disk.

And NVIDIA does make cards like you are asking for - the A100 is the fast memory offering, the A40 the bulk slower memory (though they added the 80GB A100 and did not double the A40 to 96GB so this is less true now than the P40 vs P100 gen).

Oddly, you can get close to what you are asking for with a M1 Mac Studio - 128GB of decently fast memory with a GPU that is ~0.5x a 3090 in training.

Post reply on HN