Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoGPT

github.com

Post reply on HN
Print view View original

231–240 of 334 posts

Re: NanoGPT

#231

Post by tlb »

mbrock wrote: post #77
tlb wrote: post #51

Earlier quoted context omitted.

You can't know that. Currently, 8 billion humans generate a few scientific breakthroughs per year. You'd have to run several billion ChatGPTs for a year with zero breakthroughs to have any confidence in such a claim.

With billions of GPT output streams, how do you actually discover and rank what’s significant? Screen them through some even more powerful models? I imagine it’s like a volcano eruption of text where some are absolutely brilliant and most is worthless and finding the jewels is even more demanding than generating it all.

Some theories are easily testable. For instance, ask it to write some code to efficiently solve traveling salesman problems, and then test the code on some sample problems. You can score the quality of solutions and time taken, and manually inspect the best ones.

Re: NanoGPT

#232

Post by de_nied »

karpathy wrote: post #225

Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.

Thank you for your constant contributions.

Re: NanoGPT

#233

Post by gtoubassi »

karpathy wrote: post #225

Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.

+1. I've benefited greatly from your content, e.g. your CNN lecture was incredibly accessible [0]. I still find transformers stubbornly elude my intuitions despite reading many descriptions. I would very much appreciate your video lecture on this topic.

[0] I think https://www.youtube.com/watch?v=LxfUGhug-iQ

Re: NanoGPT

#234

Post by cs702 »

karpathy wrote: post #225

Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.

Andrej: thank you!

--

To the mod (dang): IMHO Andrej's comment should probably be at the top of the page, not my comment. UPDATE: Looks like that's done. Thank you :-)

Re: NanoGPT

#235

Post by karmajuney »

wongarsu wrote: post #113
yreg wrote: post #60

Is there any trained model for text generation that you can run locally yet?

GPT2 can be run locally (on a somewhat beefy consumer GPU)

Can you add some info on what consumer GPU would be needed for this? Would a 3080 be able to handle this?

Re: NanoGPT

#236

Post by StefanWestfal »

karpathy wrote: post #225

Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.

Open accessible lectures / knowledge like yours allowed many people, me included, to turn their life around by putting in the effort and develop themselves. Thank you.

Re: NanoGPT

#237

Post by thomastjeffery »

ccozan wrote: post #68
captainmuon wrote: post #23

Earlier quoted context omitted.

I wonder about this, too. OpenAI's biggest 'moat' is that their model takes so much resources to train, not that their algorithms are particularly secret. One idea I had was to not use one single model to learn all steps of the task, but to break it up. The human brain has dedicated grammar processing parts. It is unclear whether something like a universal grammar exists, but we have at least an innate sense for rhyt…

GPT and human brain ( at least the language / speech part ) have nothing in common. We, as humans, do not use language in a generative way, is derived from a higher or very low level of abstraction ( intentions, emotions, etc ) and is explictly use for communicating something. Even this text is based on previous knowledge, saved in an abstract way, and while writing this I must follow the synthax of the language or w…

Ok top of it not having "motivation" to communicate, it has literally nothing to be communicated in the first place.

That's the key difference. We use language to express conceptualizations. We have some kind of abstract model somewhere that we are translating.

Maybe it isn't a cohesive model either. All I can say for certain is that - whatever it is - we are expressing it.

GPT does not express. It parrots. There is no conceptualization.

Re: NanoGPT

#238

Post by isoprophlex »

cs702 wrote: post #72

Andrej doesn't need to do this. He's done it because he evidently loves it, and wants to share his hard-earned knowledge with the rest of the world. He may be a product of the ivory tower, but he's been in the trenches . He knows firsthand how f-ing hard it is to ship a product. And here he is, sharing useful personal code with everyone. This github repo now has collected ~4K stars and its antecessor (minGPT) has col…

Pedantry time!

A million people building GPT models means that one in 8000 humans on earth has built one. That seems wildly off.

Linkedin has about 100.000 profiles of data scientists. Assume generously that the actual number is 10x higher. Not correcting for the fact that a data scientist isn't always a machine learning expert, etc etc, there's just no way every single one of them even KNOWS what a GPT-like model is.

Re: NanoGPT

#239

Post by minimaxir »

QuadrupleA wrote: post #217

Doesn't huggingface have dozens of freely available pretrained models like this (including various sized implementations of GPT2) and isn't the source available on most if you wanted to train them yourself? All I see in the comments is praise for the author as a person, so just wondering what's unique about this that's not available elsewhere? 730 upvotes and counting, assuming I'm missing something...

Additionally, in terms of the streamlining nanoGPT porports, HuggingFace's implementations play nice with optimization techniques such as ONNX/TensorRT, which will give you better performance than anything PyTorch-based even if minimal.

That doesn't mean an ONNX-ed nanoGPT won't be better, but the field of optimized text generation isn't as new as people claim.

Re: NanoGPT

#240

Post by isoprophlex »

QuadrupleA wrote: post #217

Doesn't huggingface have dozens of freely available pretrained models like this (including various sized implementations of GPT2) and isn't the source available on most if you wanted to train them yourself? All I see in the comments is praise for the author as a person, so just wondering what's unique about this that's not available elsewhere? 730 upvotes and counting, assuming I'm missing something...

True, but the use cases arent the same. As he did before for other models, he has a knack for distilling the code down to beautiful, self-contained examples of high didactic value.

It's an order of magnitude easier to grok the basics from this repo than from going through (admittedly more ergonomic or performant or production-ready) huggingface repos.

Post reply on HN