Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NanoGPT

github.com

Post reply on HN
Print view View original

321–330 of 334 posts

Re: NanoGPT

#321

Post by dang »

Terretta wrote: post #85

14 hours ago: https://news.ycombinator.com/item?id=34331919 Curious why HN didn't merge the submission as it usually does. Is there a "no, submit this again" option?

HN lets reposts through if the story hasn't had significant attention yet. This is to give good submissions multiple chances at getting attention. Past explanations:

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

I know it sucks when your submission was earlier and gets overlooked! We should eventually have some sort of karma-sharing to take care of this. In the meantime, it at least evens out in the long run, since the reason is randomness.

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

Re: NanoGPT

#322

Post by ironrabbit »

pms wrote: post #318
visarga wrote: post #282

Earlier quoted context omitted.

This is a didactic implementation. If you read the HuggingFace repo it is much more abstracted on account they implement many models in the same codebase. It's not fast or big, just easier to read and tweak.

If so, then why the second line of its documentation says that "it is a rewrite of minGPT that prioritizes teeth over education"?

minGPT prioritized being understandable above all else, and was not very fast. This repo includes several optimizations, but it still much more understandable than probably any other open source implementation.

Re: NanoGPT

#323

Post by m3affan »

karpathy wrote: post #225

Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.

Amazing work, much appreciated

Re: NanoGPT

#324

Post by nurettin »

karpathy wrote: post #225

Wow, fun to find this trending on HN this morning! I am currently also working on the associated video lecture (as the next episode of my video lecture series here https://karpathy.ai/zero-to-hero.html ), where I will build nanoGPT from scratch and aspire to spell everything out, as with the earlier videos. Hoping to get it out in ~2 weeks or so.

Thanks, I love your video about back propagation where you painstakingly spell out every calculation. It was like a breath of fresh air compared to other materials out there.

Re: NanoGPT

#325

Post by cactusplant7374 »

int_19h wrote: post #307
cactusplant7374 wrote: post #65

Earlier quoted context omitted.

I agree with you on this point. It’s also arguable that less people with a better education system could yield the same result with less environmental impact. But my point, poorly explained, is that whatever ChatGPT is, it isn’t original or creative thought as a human would do it. Chomsky’s example (which is based off Turing): Do submarines swim? Yes, they swim — if that’s what you mean by swimming.

We don't have any clear definitions for "creativity" to begin with. In practice, in these contexts, it seems to be defined as "whatever only humans can do" - that is, the goalposts are automatically moved with every AI advancement.

How could they be moved when they aren't even defined in the first place? Scientists don't even know where to begin when it comes to studying the mind and human consciousness.

But yes, scientists can look at your experiments and show that they don't have anything in common with human thought.

Re: NanoGPT

#326

Post by klaudioz »

Tepix wrote: post #143
arturventura wrote: post #76

This is really good, and I was really excited by it but then I read: > running on a single 8XA100 40GB node in 38 hours of training This is a $40-80k machine. Not a diss, but I would love to see an advance that would allow anyone with a high end computer to be able to improve on this model. Before that happens this whole field is going to be owned by big corporations.

If you can fit the training into 24GB, a used RTX 3090 for $700-$800 seems like a good deal at the moment. They are about 45-65% as fast as the A100 according to https://bizon-tech.com/gpu-benchmarks/NVIDIA-RTX-3090-vs-NVI... So if you buy two of these cards it will take 12-13 days instead of 38 hours but only require a $2500 PC. James Betker, who created tortoise TTS, built his own $15k machine with 8x RTX 3090 and…

Any link to the 15k machine ?. Maybe it is cheaper now.

Re: NanoGPT

#327

Post by Tepix »

klaudioz wrote: post #326
Tepix wrote: post #143

Earlier quoted context omitted.

If you can fit the training into 24GB, a used RTX 3090 for $700-$800 seems like a good deal at the moment. They are about 45-65% as fast as the A100 according to https://bizon-tech.com/gpu-benchmarks/NVIDIA-RTX-3090-vs-NVI... So if you buy two of these cards it will take 12-13 days instead of 38 hours but only require a $2500 PC. James Betker, who created tortoise TTS, built his own $15k machine with 8x RTX 3090 and…

Any link to the 15k machine ?. Maybe it is cheaper now.

I think it was a DIY machine, those RTX 3090 have gotten cheaper for sure. From my experience, going beyond 4 GPUs is a pricey affair. See [§]. All but one model of the RTX3090 require at least 3 slots.

If 4 GPUs connected via PCIe 4.0x16 are enough you can choose among various sRTX4 boards for 3000 series AMD Threadripper CPUs.

[§] https://www.reddit.com/r/deeplearning/comments/tw0olq/commen...

Another useful URL: https://www.pugetsystems.com/labs/articles/Quad-GeForce-RTX-...

Re: NanoGPT

#329

Post by themulticaster »

ben_w wrote: post #176
ccozan wrote: post #68

Earlier quoted context omitted.

GPT and human brain ( at least the language / speech part ) have nothing in common. We, as humans, do not use language in a generative way, is derived from a higher or very low level of abstraction ( intentions, emotions, etc ) and is explictly use for communicating something. Even this text is based on previous knowledge, saved in an abstract way, and while writing this I must follow the synthax of the language or w…

> and while writing this I must follow the synthax of the language or writing the right order otherwise A good example that is not, word randomised order and kombination with Mrs Spelling and fonetic spel-ing prevent ye knot that which I wrote you to komprehend. (My apologies to non-native speakers of English; if someone did that to me in German I'd have no clue what was meant). A better point is that GPT-3's trainin…

> (My apologies to non-native speakers of English; if someone did that to me in German I'd have no clue what was meant).

At least for me it's perfectly understandable (except the "Mrs" part). This reminds of those "did you know you can flip characters randomly and our brain can still understand the text" copypastas that can be found everywhere. I think it's probably quite similar for word order: As long as your sentence structure is not extremely complicated, you can probably get away with changing it any way you like. Just like nobody has issues understanding Yoda in Star Wars.

Although I think there are some limits to changing word order - I can imagine complicated legal documents might get impossible to decipher if you start randomizing word order.

Post reply on HN