Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoGPT

github.com

Post reply on HN
Print view View original

261–270 of 334 posts

Re: NanoGPT

#261

Post by randoglando »

lukebechtel wrote: post #212

I have taken several masters-level courses in Machine Learning -- and even with those credentials, I cannot recommend enough Andrej's youtube series, "Neural Networks: Zero to Hero". There, he teaches you, from scratch, how to build everything from the underlying automated gradient calculation system in pytorch, all the way up to the slower version of this model - `MinGPT`. [1] https://www.youtube.com/playlist?list=P…

How does it compare to fast.ai? As a engineer looking to learn, what should I start with?

Re: NanoGPT

#262

Post by mk_stjames »

amelius wrote: post #111
jetrink wrote: post #103

Earlier quoted context omitted.

That's a great comparison. For a real number, I just checked Runpod and you can rent a system with 8xA100 for $17/hr or ~$700 for 38 hours. Not cheap, but also pretty close to the cost of renting a premium vehicle for a few days. I've trained a few small models by renting an 1xA5000 system and that only costs $0.44/hr, which is perfect for learning and experimentation.

It would be great if a tradeoff could be made, though. For example, train at 1/10th the speed for 1/10th of the cost. This could correspond to taking public transport in your analogy, and would bring this within reach of most students.

The problem with that is currently, the available memory scales with the class of GPU.... and very large language models need 160-320GB of VRAM. So, there sadly isn't anything out there that you can load up a model this large on except a rack of 8x+ A40s/A100s.

I know there are memory channel bandwidth limits and whatnot but I really wish there was a card out there with a 3090 sized die but with 96GB of VRAM solely to make it easier to experiment with larger models. If it takes 8 days to train vs. 1, thats fine. having only two of them to get 192GB and still fit on a desk and draw normal power would be great.

Re: NanoGPT

#263

Post by marviel »

randoglando wrote: post #261
lukebechtel wrote: post #212

I have taken several masters-level courses in Machine Learning -- and even with those credentials, I cannot recommend enough Andrej's youtube series, "Neural Networks: Zero to Hero". There, he teaches you, from scratch, how to build everything from the underlying automated gradient calculation system in pytorch, all the way up to the slower version of this model - `MinGPT`. [1] https://www.youtube.com/playlist?list=P…

How does it compare to fast.ai? As a engineer looking to learn, what should I start with?

Both are good for different things.

Fast.AI is great, but it takes the top down, vs the bottom up, approach. It takes you from a production-level black box that you don't understand, down to the details. The benefit there is you get good high-level intuition of how it behaves at the "let me use this technology for a job" level.

Separately, the fast.ai library is also highly recommendable -- it comes with some state-of-the-art image recognition models, and its training wrappers are really helpful particularly for image-recognition dataset training.

Karpathy's "Neural Networks: Zero to Hero" video series starts at the level of individual neurons, and works you up to the final product. For some reason both this style, and Karpathy's conciseness appeal to me slightly more. I'm also super detail-oriented, though -- and any level of "hand waving" (even if further explanation comes later) always bothers me. He's also got some pretty high-profile industry experience which carries some weight with me.

But I'll say that both are really high-quality. -- ultimately, my recommendation would be to follow whichever one speaks most to you personally after the first 1hr or so.

EDIT: Per Jeremy's response below, if you want the bottom-up approach but like the fast.ai teaching style, you should check out "part 2" of the fast.ai set of tutorials, which is exactly that.

Re: NanoGPT

#264

Post by moralestapia »

karpathy wrote: post #225

Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.

While doing my PhD some years ago (it wasn't a PhD on AI, but very much related) I trained several models with the usual stack back then (pytorch and some others in TF). I realized that a lot of this stack could be rewritten in much simpler terms without sacrificing much fidelity and/or performance in the end.

Submissions like yours and other projects like this one (recently featured here as well) -> https://github.com/ggerganov/whisper.cpp, makes it pretty clear to me that this intuition is correct.

There's a couple tools I created back then that could push things further towards this direction, unfortunately they're not mature enough to warrant a release but the ideas they portray are worth taking a look at (IMHO) and I'll be happy to share them. If there's interest on your side (or anyone reading this thread) I'd love to talk more about it.

Re: NanoGPT

#265

Post by lossolo »

> reproduces GPT-2 (124M) on OpenWebText, running on a single 8XA100 40GB node in 38 hours of training

For comparison GPT-3 has more than 1000x more params (175B) and training time was around 2 months on ~1500 V100 GPUs which costs millions of dollars in cloud compute costs. Gopher with 280B params was trained on 4096 TPU-v3 chips, Microsoft Megatron-Turing NLG 530B trained on 2240 NVIDIA A100 cards (each card costs ~15k USD). And the most mind blowing is PaLM from Google with 540B params and trained on 6144 TPU v4, which costs around 10-30M USD in cloud compute to train.

Re: NanoGPT

#266

Post by lossolo »

visarga wrote: post #224
rsiqueira wrote: post #188

I could not find any sample (prompt and results). Can anyone provide samples of it's quality, even if it is in a narrow field of knowledge or specific use case? I tried GPT2, GPT-J 6B and GPT-NeoX 20B (implementation by Fabrice Bellard at textsynth.com/playground.html) but I could not find any production-quality scenario yet, only cherry-picked simple cases.

At this model size quality is not worth discussing. It is clearly another league from GPT-3.

Indeed, it is like comparing the speech of a 2-year-old child to that of a college professor.

Re: NanoGPT

#267

Post by thomastjeffery »

captainmuon wrote: post #249
thomastjeffery wrote: post #237

Earlier quoted context omitted.

Ok top of it not having "motivation" to communicate, it has literally nothing to be communicated in the first place. That's the key difference. We use language to express conceptualizations. We have some kind of abstract model somewhere that we are translating. Maybe it isn't a cohesive model either. All I can say for certain is that - whatever it is - we are expressing it. GPT does not express. It parrots. There is…

The more experience I get, the more I wonder if this is really the case for us. We certainly have some kind of abstract model in our heads when thinking deeply about a problem. But in many settings - in a work meeting, or socially with friends - I think it is a much more automatic process. The satisfaction you get when saying the right thing, the dread when you say something stupid: It is just like playing a game. Ma…

Intelligent but not conscious would still be a few steps ahead of GPT.

We can take a concept and refactor it symbolically. GPT can't do that. All it does is find symbols that are semantically close to other symbols.

Re: NanoGPT

#268

Post by throwawaymaths »

windexh8er wrote: post #174
arturventura wrote: post #76

This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.

But how often do you need to run this? You can run 8xA1000 on LambdaLabs [0] (no affiliation) for $8.80/hr. So you should be able to run the entire data set for less than $350. [0] https://lambdalabs.com/service/gpu-cloud#pricing

They are acknowledged at the bottom for supporting andrej's research!!

Re: NanoGPT

#269

Post by londons_explore »

ProjectArcturis wrote: post #208
londons_explore wrote: post #140

Earlier quoted context omitted.

30,000 words wouldn't be enough to train this from scratch - you'd ideally train from hundreds of millions of words at least. 30,000 words would be enough to finetune an existing model. If you did that, then the model would output text similar to the finetuning data. For example, if you finetuned it on shakespeare, then you might be able to use the model to make a new play, in shakespeare's style.

If you finetuned it on the text of Shakespeare's plays, how would it link that text to the string "Shakespeare"?

It still has the knowledge from the main training on data from across the whole internet, so would still know the word Shakespeare...

But you're right - the model finetuned on shakespeare would be good at writing a new play in the style of shakespeare, but would be bad at giving a critique of shakespeare's works.

Re: NanoGPT

#270

Post by waiseristy »

So, are there any of these projects that aren't vendor locked to NVIDIA and are able to train large models with limited GPU RAM space?

I don't mind letting my machine churn for 2-3 weeks. But I'm not looking to buy another 1000$ GPU just because CUDA is the only compute library researchers understand

Post reply on HN