Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoGPT

github.com

Post reply on HN
Print view View original

121–130 of 334 posts

Re: NanoGPT

#121

Post by belter »

wongarsu wrote: post #95
arturventura wrote: post #76

This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.

It's a $33/hour machine on AWS, so about $1250 for one training run. Not cheap, but easily in the reach of startups and educational or research institutions. Edit: or about $340 if you get the 8xA100 instance from lambdalabs, in the realm of normal hobby spending

Or $9/hour if you use Spot :-)

https://aws.amazon.com/ec2/spot/pricing/

Re: NanoGPT

#122

Post by code_runner »

justusthane wrote: post #107

This is a dumb question about language models in general, not necessarily specific to NanoGPT: why is all the focus on training? Can I download and run a pre-trained model locally? Surely the specs required to run a model are much, much lower than those required to train the model?

I believe the training is where the architecture of the model is most apparent. You can absolutely download plenty of pre-trained models.

You will also probably need to fine tune for a specific use case, so a common approach is downloading a pre-trained model and fine tuning.

I think including the “from scratch” tuning script is educational more than anything else.

Re: NanoGPT

#123

Post by buzzdenver »

For an AI noob like me: can you use spot instances to train models? They are about 1/3rd the price on AWS compared to on demand ones, so it'd make a significant difference.

Re: NanoGPT

#124

Post by londons_explore »

latexr wrote: post #84
cs702 wrote: post #72

Andrej doesn't need to do this. He's done it because he evidently loves it, and wants to share his hard-earned knowledge with the rest of the world. He may be a product of the ivory tower, but he's been in the trenches . He knows firsthand how f-ing hard it is to ship a product. And here he is, sharing useful personal code with everyone. This github repo now has collected ~4K stars and its antecessor (minGPT) has col…

Edit: the OP has updated their wording to make it clear they meant any kind of viewing or usage. I don’t think any of us would disagree more people use code than star repos. Original comment left below with original quote, since this has gotten a number of replies that would stop making sense with a larger edit. > Normally, the number of people who clone or copy code from a repo is one to two orders of magnitude larg…

Some repos have code that 'phones home' when run. For example, checking for updates or security vulnerabilities.

By checking the usage statistics on that server, you can get an idea how many users there are, and typically it's far higher than the number of stars.

Re: NanoGPT

#125

Post by latexr »

chirau wrote: post #116
latexr wrote: post #84

Earlier quoted context omitted.

Edit: the OP has updated their wording to make it clear they meant any kind of viewing or usage. I don’t think any of us would disagree more people use code than star repos. Original comment left below with original quote, since this has gotten a number of replies that would stop making sense with a larger edit. > Normally, the number of people who clone or copy code from a repo is one to two orders of magnitude larg…

How many projects have you starred and how many have you cloned? Whilst starring is simpler, the incentive is much lower than that of cloning. Especially for projects you just want to use and not contribute to or follow. In my many years of work, i have only starred less than 50 repos. I am sure i have cloned more than a thousand.

> How many projects have you starred and how many have you cloned?

I seldom star, but neither you nor I can be extrapolated to the general community. I have thousands of stars in some repos, and I know a significant number of those users don’t even code, let alone clone repos or copy code, they’re interested in the final product. They have GitHub accounts because it’s the way to report bugs or make feature requests.

The OP made a claim. All I’m looking to understand is if it has data backing it up or it’s just a gut feeling, because if it’s the former I’ll have learned something and made a correction of my mental model of the world. Sharing more anecdotes will leave us stuck in the same situation.

Re: NanoGPT

#126

Post by londons_explore »

amelius wrote: post #111
jetrink wrote: post #103

Earlier quoted context omitted.

That's a great comparison. For a real number, I just checked Runpod and you can rent a system with 8xA100 for $17/hr or ~$700 for 38 hours. Not cheap, but also pretty close to the cost of renting a premium vehicle for a few days. I've trained a few small models by renting an 1xA5000 system and that only costs $0.44/hr, which is perfect for learning and experimentation.

It would be great if a tradeoff could be made, though. For example, train at 1/10th the speed for 1/10th of the cost. This could correspond to taking public transport in your analogy, and would bring this within reach of most students.

Slower training tends to be only a little cheaper, because most modern architectures parallelize well, and they just care about the number of flops.

If you want to reduce cost, you need to reduce the model size, and you'll get worse results for less money.

Re: NanoGPT

#127

Post by londons_explore »

kzrdude wrote: post #110
arturventura wrote: post #76

This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.

How are universities and colleges dealing with this kind of demand for computing power? It must be hard to be able to do some courses now.

'group project'

Re: NanoGPT

#128

Post by belter »

buzzdenver wrote: post #123

For an AI noob like me: can you use spot instances to train models? They are about 1/3rd the price on AWS compared to on demand ones, so it'd make a significant difference.

Yes you can. In Oregon you could eventually get this instance at $9. I say eventually, because of course Spot allocation is not guaranteed. ( And neither is On Demand ...but that is a story for another day)

https://aws.amazon.com/ec2/spot/pricing/

Re: NanoGPT

#129

Post by swader999 »

pavlov wrote: post #89
arturventura wrote: post #76

This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.

I don't know if that's a blocker. Ordinary people commonly rent a $40k machine for 38 hours from companies like Avis and Hertz. If training a large model now costs the same as driving to visit grandma, that seems like a pretty good deal.

You have to gas it up and heaven help you if it gets a scratch or a scuff.

Re: NanoGPT

#130

Post by swader999 »

Would it be possible to take all my user manuals and past customer Q&A and train on just on that to produce a customer helper chat bot?
Post reply on HN