Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoGPT

github.com

Post reply on HN
Print view View original

291–300 of 334 posts

Re: NanoGPT

#291

Post by jph00 »

marviel wrote: post #263
randoglando wrote: post #261

Earlier quoted context omitted.

How does it compare to fast.ai? As a engineer looking to learn, what should I start with?

Both are good for different things. Fast.AI is great, but it takes the top down, vs the bottom up, approach. It takes you from a production-level black box that you don't understand, down to the details. The benefit there is you get good high-level intuition of how it behaves at the "let me use this technology for a job" level. Separately, the fast.ai library is also highly recommendable -- it comes with some state-o…

fast.ai has both - the "part 1" section is top-down, and the "part 2" section is bottom up. You can do part 2 without having done part 1. Part 2 starts with implementing matrix multiplication from scratch, then backprop from scratch, then SGD from scratch, etc.

There will be a new version of the part 2 course out in a few weeks. It even covers stuff like random number generation from scratch, convolutions from scratch, etc. It gradually works all the way up to Stable Diffusion.

@karpathy's and the fast.ai lessons work well together. They cover similar topics from different angles.

(I'm the primary creator of the fast.ai courses.)

Re: NanoGPT

#292

Post by wongarsu »

karmajuney wrote: post #235
wongarsu wrote: post #113

Earlier quoted context omitted.

GPT2 can be run locally (on a somewhat beefy consumer GPU)

Can you add some info on what consumer GPU would be needed for this? Would a 3080 be able to handle this?

Assuming you get the 12GB version of the 3080. A 2080TI is another option. Though you can reduce precision or use one of the smaller GPT2 versions to run on smaller cards as well.

Re: NanoGPT

#293

Post by imranq »

karpathy wrote: post #225

Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.

Just wanted to say thank you for all the incredible work and resources you publish. I've lost track of all the different skills I've learned from you, from computer vision, RNNs, minGPT, even speedcubing :D

Re: NanoGPT

#294

Post by jph00 »

arturventura wrote: post #76

This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.

A couple of weeks ago a new paper came out that shows how to train a high quality language model on a single GPU in one day.

https://arxiv.org/abs/2212.14034

Re: NanoGPT

#295

Post by marviel »

jph00 wrote: post #291
marviel wrote: post #263

Earlier quoted context omitted.

Both are good for different things. Fast.AI is great, but it takes the top down, vs the bottom up, approach. It takes you from a production-level black box that you don't understand, down to the details. The benefit there is you get good high-level intuition of how it behaves at the "let me use this technology for a job" level. Separately, the fast.ai library is also highly recommendable -- it comes with some state-o…

fast.ai has both - the "part 1" section is top-down, and the "part 2" section is bottom up. You can do part 2 without having done part 1. Part 2 starts with implementing matrix multiplication from scratch, then backprop from scratch, then SGD from scratch, etc. There will be a new version of the part 2 course out in a few weeks. It even covers stuff like random number generation from scratch, convolutions from scratc…

That's awesome! I did not know that part 2 was structured this way, and will check it out. Will be really neat to see you teach stable diffusion.

Thanks for your work on fast.ai!

Re: NanoGPT

#296

Post by misza222 »

karpathy wrote: post #225

Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.

Thanks for your work Andrej! I've been doing earlier lectures and this is absolutely fantastic educational content!

Re: NanoGPT

#297

Post by haldujai »

arturventura wrote: post #76

This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.

If you can’t fit the model on your resources you can leverage DeepSpeed’s ZeRO-offload which will let you train GPT2 on a single V100 (32gb).

Alternatively, if you’re researching (with the caveat that you have to either publish, open source or share your results in a blog post) you can also get access to Google’s TPU research cloud which gives you a few v3-8s for 30 days (can’t do distributed training on devices but can run workloads in parallel). You can also ask nicely for a pod, I’ve been granted access to a v3-32 for 14 days pretty trivially which (if optimized) has more throughput than 8xA100 on transformer models.

TPUs and moreso pods are a bit harder to work with and TF performs far better than PyTorch on them.

https://www.deepspeed.ai/tutorials/zero-offload/

https://medium.com/analytics-vidhya/googles-tpu-research-clo...

Re: NanoGPT

#298

Post by haldujai »

lossolo wrote: post #281
renewiltord wrote: post #273

Earlier quoted context omitted.

What's the largest language model I can run on a 3090 with 24 GiB RAM?

Depends on precision, you can run ~5B model with fp32 precision or ~11B fp16 model max. Int8 is really bad for real world use case so not mentioning it. But if you are looking to get performance of ChatGPT or GPT-3 then don't waste your time, all GPT-3 like small LLM models (below at least 60B params) are useless for any real world use case, they are just toys.

If you specifically mean a general LLM trained on a general language corpus with instruction finetuning this is correct.

Fortunately very few real world use cases need to be this general.

If you are training a LLM on a domain specific corpus or finetuning on specific downstream tasks even relatively tiny models at 330m params are definitely useful and not “toys” and can be used to accurately perform tasks such as semantic text search, document summarization and named entity recognition.

Re: NanoGPT

#299

Post by Gigachad »

anilshanbhag wrote: post #219
arturventura wrote: post #76

This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.

If GPT-2 / nanoGPT needs this setup, just imagine what GPT3 / chatGPT needs!

Supposedly even running the trained model for ChatGPT is extremely expensive unlike the image generators which can largely be run on a consumer device.

Re: NanoGPT

#300

Post by renewiltord »

lossolo wrote: post #281
renewiltord wrote: post #273

Earlier quoted context omitted.

What's the largest language model I can run on a 3090 with 24 GiB RAM?

Depends on precision, you can run ~5B model with fp32 precision or ~11B fp16 model max. Int8 is really bad for real world use case so not mentioning it. But if you are looking to get performance of ChatGPT or GPT-3 then don't waste your time, all GPT-3 like small LLM models (below at least 60B params) are useless for any real world use case, they are just toys.

Okay, thank you. Perfect response.
Post reply on HN