Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoGPT

github.com

Post reply on HN
Print view View original

131–140 of 334 posts

Re: NanoGPT

#131

Post by Metus »

joaogui1 wrote: post #81
captainmuon wrote: post #23

Earlier quoted context omitted.

I wonder about this, too. OpenAI's biggest 'moat' is that their model takes so much resources to train, not that their algorithms are particularly secret. One idea I had was to not use one single model to learn all steps of the task, but to break it up. The human brain has dedicated grammar processing parts. It is unclear whether something like a universal grammar exists, but we have at least an innate sense for rhyt…

This biggest most is high-quality data. Both their proprietary datasets (WebText, WebText2 etc), but also now their human-annotated data. Another secondary moat is their expertise with training models using PPO (their RL method), they can get results that are quite better than other labs. I say this moat is secondary because it's possible that you can get similar results with other RL algorithms (e.g. DeepMind using…

I find OpenAI having exclusive access to that kind of high-quality data more concerning than them having access to their current amount of compute and currently trained model. A couple of million dollars worth of compute is in the realm of any medium sized research university, larger company or any country worth of mention. And seeing as Moore's law still applies to GPU, the cost will only fall.

However high-quality data is scarce. I would be willing to fund a proper effort to create high-quality data.

Re: NanoGPT

#132

Post by eismcc »

Terretta wrote: post #85

14 hours ago: https://news.ycombinator.com/item?id=34331919 Curious why HN didn't merge the submission as it usually does. Is there a "no, submit this again" option?

The other post probably didn’t make it to the front page

Re: NanoGPT

#133

Post by naasking »

jamesfisher wrote: post #69

For casual readers like me: are there examples of what this can do once trained? E.g. it mentions training on Shakespeare, but gives no examples of fake Shakespearean.

The repo seems to imply that it matches GPT-2, so I imagine any analyses of GPT-2 will give you a good idea.

Re: NanoGPT

#134

Post by tomComb »

cs702 wrote: post #72

Andrej doesn't need to do this. He's done it because he evidently loves it, and wants to share his hard-earned knowledge with the rest of the world. He may be a product of the ivory tower, but he's been in the trenches . He knows firsthand how f-ing hard it is to ship a product. And here he is, sharing useful personal code with everyone. This github repo now has collected ~4K stars and its antecessor (minGPT) has col…

Him doing this is not like when your average bloke does it.

He appears to be building a business and maintaining his profile. And there is nothing wrong with that - I admire him for for pursuing his career in this positive and helpful way.

But random folks do this sort of thing everyday with no such career goals and little recognition, so I'm not sure it is this specific contribution that needs to be called out.

Re: NanoGPT

#135

Post by latexr »

londons_explore wrote: post #124
latexr wrote: post #84

Earlier quoted context omitted.

Edit: the OP has updated their wording to make it clear they meant any kind of viewing or usage. I don’t think any of us would disagree more people use code than star repos. Original comment left below with original quote, since this has gotten a number of replies that would stop making sense with a larger edit. > Normally, the number of people who clone or copy code from a repo is one to two orders of magnitude larg…

Some repos have code that 'phones home' when run. For example, checking for updates or security vulnerabilities. By checking the usage statistics on that server, you can get an idea how many users there are, and typically it's far higher than the number of stars.

That just tells us that more people use the code than star the repo. I don’t think that’d be a surprise to anyone. The claim was that more people clone and copy code from the repo than the ones who star it, which is a different matter from the number of users.

Re: NanoGPT

#136

Post by yreg »

buzzdenver wrote: post #123

For an AI noob like me: can you use spot instances to train models? They are about 1/3rd the price on AWS compared to on demand ones, so it'd make a significant difference.

Why not? This is the exact use case of what Spot instances seem to be for. (Not hosting a service, but just calculating something for yourself.)

Re: NanoGPT

#137

Post by sharemywin »

To me this is the important quote:

Unlike OpenWebText this will run in seconds. Finetuning takes very little time, e.g. on a single GPU just a few minutes. Run an example finetuning like:

Re: NanoGPT

#138

Post by londons_explore »

nprateem wrote: post #117

If I trained this on a 30,000 word document could it give me a summary? Or would there be no need to train it in that case, and I could just tell it "Summarise this: "?

The context window (block size) of this model is 1024 symbols. Symbols approximately map to words. So you can't ask it to summarize anything over 1024 words.

Re: NanoGPT

#139

Post by naasking »

ccozan wrote: post #68
captainmuon wrote: post #23

Earlier quoted context omitted.

I wonder about this, too. OpenAI's biggest 'moat' is that their model takes so much resources to train, not that their algorithms are particularly secret. One idea I had was to not use one single model to learn all steps of the task, but to break it up. The human brain has dedicated grammar processing parts. It is unclear whether something like a universal grammar exists, but we have at least an innate sense for rhyt…

GPT and human brain ( at least the language / speech part ) have nothing in common. We, as humans, do not use language in a generative way, is derived from a higher or very low level of abstraction ( intentions, emotions, etc ) and is explictly use for communicating something. Even this text is based on previous knowledge, saved in an abstract way, and while writing this I must follow the synthax of the language or w…

These are conceptual "differences" that don't actually explain the mechanics of what's going on. For all you know "motivation", "intentions", etc. are also just GPT-like subsystems, in which case the underlying mechanics are not as different as you imply.

Re: NanoGPT

#140

Post by londons_explore »

nprateem wrote: post #117

If I trained this on a 30,000 word document could it give me a summary? Or would there be no need to train it in that case, and I could just tell it "Summarise this: "?

30,000 words wouldn't be enough to train this from scratch - you'd ideally train from hundreds of millions of words at least.

30,000 words would be enough to finetune an existing model. If you did that, then the model would output text similar to the finetuning data. For example, if you finetuned it on shakespeare, then you might be able to use the model to make a new play, in shakespeare's style.

Post reply on HN