151–160 of 334 posts
Earlier quoted context omitted.
I don't know if that's a blocker. Ordinary people commonly rent a $40k machine for 38 hours from companies like Avis and Hertz. If training a large model now costs the same as driving to visit grandma, that seems like a pretty good deal.
You have to gas it up and heaven help you if it gets a scratch or a scuff.
Andrej doesn't need to do this. He's done it because he evidently loves it, and wants to share his hard-earned knowledge with the rest of the world. He may be a product of the ivory tower, but he's been in the trenches . He knows firsthand how f-ing hard it is to ship a product. And here he is, sharing useful personal code with everyone. This github repo now has collected ~4K stars and its antecessor (minGPT) has col…
Him doing this is not like when your average bloke does it. He appears to be building a business and maintaining his profile. And there is nothing wrong with that - I admire him for for pursuing his career in this positive and helpful way. But random folks do this sort of thing everyday with no such career goals and little recognition, so I'm not sure it is this specific contribution that needs to be called out.
I go the other way. I would like to thank anyone who releases open source code, whether they cause big ripples or not.
If I trained this on a 30,000 word document could it give me a summary? Or would there be no need to train it in that case, and I could just tell it "Summarise this: "?
The context window (block size) of this model is 1024 symbols. Symbols approximately map to words. So you can't ask it to summarize anything over 1024 words.
Earlier quoted context omitted.
Could this be distributed? Put all those mining GPUs to work. A lot of people like participating in public projects like this. I would!
>> GPT-3 took 355 years to train > Could this be distributed? Put all those mining GPUs to work. Nope. It's a strictly O(n) process. If it weren't for the foresight of George Patrick Turnbull in 1668, we would not be anywhere close to these amazing results today.
Earlier quoted context omitted.
I checked my 5 year old repository of ~300 stars. It has a ~100 unique clones a month. So if the average was half of it then the 1 order of magnitude would be quite an accurate approximation. I think the biggest difference with a clone and a star is that a star requires an account and some vested interest in the social network of Github. Anyone who is not interested in the social aspect can just bookmark it. I guess…
Thank you for sharing your anecdata. In my experience, the number of clones per month is much higher at first, and then decays gradually until it settles into a stable run-rate, so it's likely that you've had more than 100 x 12 x 5 clones over those five years -- i.e., between one and two orders of magnitude the number of stars, 300.
(This is a tool that most people install and run without any interaction with GitHub, since it is in package managers)
Earlier quoted context omitted.
You have to gas it up and heaven help you if it gets a scratch or a scuff.
Great news! Cloud instances energy usage is included in their price, and because they're remote and transient it's impossible to permanently damage them.
For casual readers like me: are there examples of what this can do once trained? E.g. it mentions training on Shakespeare, but gives no examples of fake Shakespearean.
The repo seems to imply that it matches GPT-2, so I imagine any analyses of GPT-2 will give you a good idea.
Earlier quoted context omitted.
GPT and human brain ( at least the language / speech part ) have nothing in common. We, as humans, do not use language in a generative way, is derived from a higher or very low level of abstraction ( intentions, emotions, etc ) and is explictly use for communicating something. Even this text is based on previous knowledge, saved in an abstract way, and while writing this I must follow the synthax of the language or w…
These are conceptual "differences" that don't actually explain the mechanics of what's going on. For all you know "motivation", "intentions", etc. are also just GPT-like subsystems, in which case the underlying mechanics are not as different as you imply.
Earlier quoted context omitted.
The context window (block size) of this model is 1024 symbols. Symbols approximately map to words. So you can't ask it to summarize anything over 1024 words.
Yeah that's the issue I was thinking of, how to get it to summarise large documents. Has anyone any ideas?
Divide your 30,000 word document into a hundred 300 word chuncks. For each chunk, give as input:
Please summarize the following text into 50 words:
[chunk]
You can improve the results by doing the process again, but this time giving some context:
Please summarize the following text, an extract of a document about [1st attempt at a summary], into 50 words:
[chunk]This is a dumb question about language models in general, not necessarily specific to NanoGPT: why is all the focus on training? Can I download and run a pre-trained model locally? Surely the specs required to run a model are much, much lower than those required to train the model?
Also you can perform "fine tuning" which means you start with a trained model and train it further on your own data, allowing you to customize the model for specific tasks.