241–250 of 334 posts
To train small gpt-like models, there's also aitextgen: https://github.com/minimaxir/aitextgen
I still have a few ideas there (including another secret approach at better text generation) but it's hard to determine ROI.
Earlier quoted context omitted.
It's a $33/hour machine on AWS, so about $1250 for one training run. Not cheap, but easily in the reach of startups and educational or research institutions. Edit: or about $340 if you get the 8xA100 instance from lambdalabs, in the realm of normal hobby spending
Or $9/hour if you use Spot :-) https://aws.amazon.com/ec2/spot/pricing/
Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.
Andrej doesn't need to do this. He's done it because he evidently loves it, and wants to share his hard-earned knowledge with the rest of the world. He may be a product of the ivory tower, but he's been in the trenches . He knows firsthand how f-ing hard it is to ship a product. And here he is, sharing useful personal code with everyone. This github repo now has collected ~4K stars and its antecessor (minGPT) has col…
This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.
How are universities and colleges dealing with this kind of demand for computing power? It must be hard to be able to do some courses now.
These machines are pretty much crunching numbers 24/7, and your project will get appended to a queue.
Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.
Earlier quoted context omitted.
At this point there is no framework that suggests GPT understands the underlying data. It can’t assign meaning as a human would. It can’t consume hundreds of math textbooks and learn the principles of math and then apply them more broadly to science textbooks and research papers. It can’t even reliably add two numbers. Yes, brute forcing with hard AI can produce many thoughts. But the AI wouldn’t know they are correc…
> At this point there is no framework that suggests GPT understands the underlying data. It can’t assign meaning as a human would. Actually there are many indications that GPT understands the data, because its output mostly makes sense. The reason it can't assign meaning the way a human would is because a human can correlate words with other sensory data that GPT doesn't have access to. That's where GPT creates nonse…
Earlier quoted context omitted.
GPT and human brain ( at least the language / speech part ) have nothing in common. We, as humans, do not use language in a generative way, is derived from a higher or very low level of abstraction ( intentions, emotions, etc ) and is explictly use for communicating something. Even this text is based on previous knowledge, saved in an abstract way, and while writing this I must follow the synthax of the language or w…
Ok top of it not having "motivation" to communicate, it has literally nothing to be communicated in the first place. That's the key difference. We use language to express conceptualizations. We have some kind of abstract model somewhere that we are translating. Maybe it isn't a cohesive model either. All I can say for certain is that - whatever it is - we are expressing it. GPT does not express. It parrots. There is…
But you are of course right with GPT, it has no inner life and only parrots. It completely lacks something like an inner state, an existence outside of the brief moment it is invoked, or anything like reflection. Reminds me of the novel "Blindsight" (which I actually haven't read yet, but heard good things about!) where there are beings that are intelligent, but not conscious.
To train small gpt-like models, there's also aitextgen: https://github.com/minimaxir/aitextgen
As the creator of aitextgen, I'm mixed on continuing support since there doesn't seem to be as much demand as expected for small GPT models given the success and cost-effectiveness of GPT-3/ChatGPT, unfortunately. I still have a few ideas there (including another secret approach at better text generation) but it's hard to determine ROI.
Not everyone is trying to replicate CHATGPT results for certain tasks.