Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoGPT

github.com

Post reply on HN
Print view View original

111–120 of 334 posts

Re: NanoGPT

#111

Post by amelius »

jetrink wrote: post #103
pavlov wrote: post #89

Earlier quoted context omitted.

I don't know if that's a blocker. Ordinary people commonly rent a $40k machine for 38 hours from companies like Avis and Hertz. If training a large model now costs the same as driving to visit grandma, that seems like a pretty good deal.

That's a great comparison. For a real number, I just checked Runpod and you can rent a system with 8xA100 for $17/hr or ~$700 for 38 hours. Not cheap, but also pretty close to the cost of renting a premium vehicle for a few days. I've trained a few small models by renting an 1xA5000 system and that only costs $0.44/hr, which is perfect for learning and experimentation.

It would be great if a tradeoff could be made, though. For example, train at 1/10th the speed for 1/10th of the cost.

This could correspond to taking public transport in your analogy, and would bring this within reach of most students.

Re: NanoGPT

#112

Post by weird-eye-issue »

wongarsu wrote: post #95
arturventura wrote: post #76

This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.

It's a $33/hour machine on AWS, so about $1250 for one training run. Not cheap, but easily in the reach of startups and educational or research institutions. Edit: or about $340 if you get the 8xA100 instance from lambdalabs, in the realm of normal hobby spending

After training don't you have to keep it running if you want to use it?

Re: NanoGPT

#114

Post by imranq »

I would love to see a minInstructGPT or a minRetro, or maybe something that combines instruction and retrieval into a readable codebase!

Re: NanoGPT

#115

Post by wongarsu »

weird-eye-issue wrote: post #112
wongarsu wrote: post #95

Earlier quoted context omitted.

It's a $33/hour machine on AWS, so about $1250 for one training run. Not cheap, but easily in the reach of startups and educational or research institutions. Edit: or about $340 if you get the 8xA100 instance from lambdalabs, in the realm of normal hobby spending

After training don't you have to keep it running if you want to use it?

Just download the model and run it on something much smaller and cheaper. Bigger models like GPT-J are a bit of a pain to run, but GPT2-sized models run just fine on consumer GPUs.

Re: NanoGPT

#116

Post by chirau »

latexr wrote: post #84
cs702 wrote: post #72

Andrej doesn't need to do this. He's done it because he evidently loves it, and wants to share his hard-earned knowledge with the rest of the world. He may be a product of the ivory tower, but he's been in the trenches . He knows firsthand how f-ing hard it is to ship a product. And here he is, sharing useful personal code with everyone. This github repo now has collected ~4K stars and its antecessor (minGPT) has col…

Edit: the OP has updated their wording to make it clear they meant any kind of viewing or usage. I don’t think any of us would disagree more people use code than star repos. Original comment left below with original quote, since this has gotten a number of replies that would stop making sense with a larger edit. > Normally, the number of people who clone or copy code from a repo is one to two orders of magnitude larg…

How many projects have you starred and how many have you cloned?

Whilst starring is simpler, the incentive is much lower than that of cloning. Especially for projects you just want to use and not contribute to or follow.

In my many years of work, i have only starred less than 50 repos. I am sure i have cloned more than a thousand.

Re: NanoGPT

#117

Post by nprateem »

If I trained this on a 30,000 word document could it give me a summary? Or would there be no need to train it in that case, and I could just tell it "Summarise this: "?

Re: NanoGPT

#118

Post by [deleted] »

sjadoinqwoeihad wrote: post #102
latexr wrote: post #84

Earlier quoted context omitted.

Edit: the OP has updated their wording to make it clear they meant any kind of viewing or usage. I don’t think any of us would disagree more people use code than star repos. Original comment left below with original quote, since this has gotten a number of replies that would stop making sense with a larger edit. > Normally, the number of people who clone or copy code from a repo is one to two orders of magnitude larg…

I checked my 5 year old repository of ~300 stars. It has a ~100 unique clones a month. So if the average was half of it then the 1 order of magnitude would be quite an accurate approximation. I think the biggest difference with a clone and a star is that a star requires an account and some vested interest in the social network of Github. Anyone who is not interested in the social aspect can just bookmark it. I guess…

[deleted]

Re: NanoGPT

#120

Post by QuesnayJr »

awestroke wrote: post #2

Are there any possible technologal or scientific leaps on the horizon that would reduce training time by an order of magnitude or more? GPT-3 took 355 years to train with incredibly expensive hardware, which means small players have no chance to push the state of the art

There are a couple of cases where small changes in the model make training much quicker. For example, the currently leading Go AI, KataGo, requires much less time to train than AlphaGo did.
Post reply on HN