Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoGPT

github.com

Post reply on HN
Print view View original

201–210 of 334 posts

Re: NanoGPT

#201

Post by vanpelt »

bilsbie wrote: post #165

Really cool. Can anyone answer these questions: Should I use this or minGPT? It says it needs 8XA100 40GB node. What is that and where do I acquire it? Could someone else train this and then send me the model? What would be required to run it as opposed to training it?

A100’s are Nvidia GPU’s. You can rent them from providers like AWS or LamdaLabs. The readme has instructions for downloading the original GPT2 weights from OpenAI. You can also train a very simple version on a smaller dataset from your laptop as described in the README.

If you just want to play with a similar but much better model goto https://chat.openai.com

Re: NanoGPT

#202

Post by ausbah »

justusthane wrote: post #107

This is a dumb question about language models in general, not necessarily specific to NanoGPT: why is all the focus on training? Can I download and run a pre-trained model locally? Surely the specs required to run a model are much, much lower than those required to train the model?

inference can still be a bottleneck i think since you usually load the whole thing into memory which is 32-64GB+ usually?

Re: NanoGPT

#203

Post by generalizations »

londons_explore wrote: post #190
londons_explore wrote: post #159

Earlier quoted context omitted.

People have had some success with the following process: Divide your 30,000 word document into a hundred 300 word chuncks. For each chunk, give as input: Please summarize the following text into 50 words: [chunk] Join all the outputs together, and you now have a shorter document. Repeat the process recursively. You can improve the results by doing the process again, but this time giving some context: Please summarize…

You can also use "Please suggest a section title for the following text". Then that title can be used in the 2nd round, for example using a query of the form "The following is an extract from the Introduction section of a document about The benefits and disadvantages of nuclear power in sweden :"

I imagine you could do even better by finetuning the neural net on the document before asking for the recursive summary. Then it has all the information to work with, albeit in a compressed form.

Re: NanoGPT

#204

Post by modeless »

tomComb wrote: post #134
cs702 wrote: post #72

Andrej doesn't need to do this. He's done it because he evidently loves it, and wants to share his hard-earned knowledge with the rest of the world. He may be a product of the ivory tower, but he's been in the trenches . He knows firsthand how f-ing hard it is to ship a product. And here he is, sharing useful personal code with everyone. This github repo now has collected ~4K stars and its antecessor (minGPT) has col…

Him doing this is not like when your average bloke does it. He appears to be building a business and maintaining his profile. And there is nothing wrong with that - I admire him for for pursuing his career in this positive and helpful way. But random folks do this sort of thing everyday with no such career goals and little recognition, so I'm not sure it is this specific contribution that needs to be called out.

What business is he building?

Re: NanoGPT

#205

Post by ProjectArcturis »

arturventura wrote: post #76

This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.

That's to train it from scratch, though, right? If you preload the GPT2 weights you don't need to do this. You can just give it additional training on your texts.

Re: NanoGPT

#206

Post by mcbuilder »

amelius wrote: post #111
jetrink wrote: post #103

Earlier quoted context omitted.

That's a great comparison. For a real number, I just checked Runpod and you can rent a system with 8xA100 for $17/hr or ~$700 for 38 hours. Not cheap, but also pretty close to the cost of renting a premium vehicle for a few days. I've trained a few small models by renting an 1xA5000 system and that only costs $0.44/hr, which is perfect for learning and experimentation.

It would be great if a tradeoff could be made, though. For example, train at 1/10th the speed for 1/10th of the cost. This could correspond to taking public transport in your analogy, and would bring this within reach of most students.

Well if it used to cost you $1 for 1hr at 1x speed, now it will take you 10hr at 0.1x speed, and if my math checks out $1. You need to shrink the model.

Re: NanoGPT

#207

Post by mrg3_2013 »

cs702 wrote: post #72

Andrej doesn't need to do this. He's done it because he evidently loves it, and wants to share his hard-earned knowledge with the rest of the world. He may be a product of the ivory tower, but he's been in the trenches . He knows firsthand how f-ing hard it is to ship a product. And here he is, sharing useful personal code with everyone. This github repo now has collected ~4K stars and its antecessor (minGPT) has col…

Thoughtful post! Everything so true! I am always amazed by individuals who truly are educators of the world.

Re: NanoGPT

#208

Post by ProjectArcturis »

londons_explore wrote: post #140
nprateem wrote: post #117

If I trained this on a 30,000 word document could it give me a summary? Or would there be no need to train it in that case, and I could just tell it "Summarise this: "?

30,000 words wouldn't be enough to train this from scratch - you'd ideally train from hundreds of millions of words at least. 30,000 words would be enough to finetune an existing model. If you did that, then the model would output text similar to the finetuning data. For example, if you finetuned it on shakespeare, then you might be able to use the model to make a new play, in shakespeare's style.

If you finetuned it on the text of Shakespeare's plays, how would it link that text to the string "Shakespeare"?

Re: NanoGPT

#209

Post by eismcc »

mittermayr wrote: post #22

As someone who's been in software for almost 25 years now, I read through this in amazement of how much new stuff still keeps coming in. This industry never stops and that makes it such a fascinating (but arguably harsh) world to be in. Looking at this feels like seeing the source code of a 64k demo, learning about Mode 13h and trying to replicate it in Turbo Pascal. And, much like the old days of graphics programmin…

It will be funny to look back from the future and think, wow, how did we get anything done with only 40GB RAM

Re: NanoGPT

#210

Post by DesiLurker »

speed_spread wrote: post #151
swader999 wrote: post #129

Earlier quoted context omitted.

You have to gas it up and heaven help you if it gets a scratch or a scuff.

Great news! Cloud instances energy usage is included in their price, and because they're remote and transient it's impossible to permanently damage them.

but you still have to pay for network ingress/egress traffic.
Post reply on HN