Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

github.com

Post reply on HN
Print view View original

21–30 of 62 posts

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#21

Post by LeanderK »

MisterBastahrd wrote: post #17
up2isomorphism wrote: post #12

Earlier quoted context omitted.

> For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI). To me this is a downside compared to the NPC generated by humans, since that’s the only reason I would like to read them.

You don't even necessarily need to have them coming up with valid speech. Simply giving out random quests and rewards would keep people running on a loot treadmill for most open world multiplayer games.

I think at first even background NPCs that don't give quests and rewards would be nice. Sort of give everyone a character and let them just babble. It breaks the immersion to hear the same phrases repeatedly. You can still handcraft every important quest and interaction to achieve high fidelity, but I would like random NPCs you bump into to not just repeat things all the time.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#22

Post by ImHereToVote »

swatcoder wrote: post #18
tomalaci wrote: post #7

With how much NVidia is developing AI-workload accelerating hardware, I expect this will cost maybe few dozen dollars and train in few hours within next few years. What I think will be interesting is when commodity hardware can run cheap inference from very capable, specialized models. Pretty sure it will spawn a new golden age of AI-powered desktop applications. For example, video game space has already been trying…

> For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI). This'll be a niche for a long, long time. Games are generally carefully crafted to deliver a specific mechanical and/or narrative experience. A world populated by LLM/etc bots or content is one choice of what that experience might be, but it's not going to be a very satisfying one f…

I don't have needs. I have wants. You see, I don't need. I want.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#23

Post by mmoskal »

jamestimmins wrote: post #15
michaelmior wrote: post #9

Earlier quoted context omitted.

Probably not with the same amount of training time, but I'd imagine a recent MBP GPU could handle GPT-2 training. The biggest challenge is that the training would need to be reimplemented for Metal instead of CUDA.

Ah so I couldn't just run this on my laptop for ~48 hours? That's too bad.

H100 SXM is 2000 TFLOPS at FP16. Multiply by 8.

M3 Max is 28 TFLOPS at FP16.

Based on FLOPS alone, it would be more like a year or two.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#24

Post by swatcoder »

MisterBastahrd wrote: post #17
up2isomorphism wrote: post #12

Earlier quoted context omitted.

> For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI). To me this is a downside compared to the NPC generated by humans, since that’s the only reason I would like to read them.

You don't even necessarily need to have them coming up with valid speech. Simply giving out random quests and rewards would keep people running on a loot treadmill for most open world multiplayer games.

To make this work, you need your LLM-based AI to outperform any other form of generating quests and rewards -- and that performance is measured on things like player enjoyment, game/story progression, exploitability, client system requirements or server operating costs, etc and most of those things are very hard to constrain or optimize for with an LLM right now.

While the costs are hidden from end users and are going down quickly, good LLM's remain very expensive to run and very hard to keep on track compared to other options.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#25

Post by Davidzheng »

mmoskal wrote: post #23
jamestimmins wrote: post #15

Earlier quoted context omitted.

Ah so I couldn't just run this on my laptop for ~48 hours? That's too bad.

H100 SXM is 2000 TFLOPS at FP16. Multiply by 8. M3 Max is 28 TFLOPS at FP16. Based on FLOPS alone, it would be more like a year or two.

Can you estimate how long it would take to replicate alphago zero today on one set of 8xH100.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#28

Post by robbomacrae »

tomalaci wrote: post #7

With how much NVidia is developing AI-workload accelerating hardware, I expect this will cost maybe few dozen dollars and train in few hours within next few years. What I think will be interesting is when commodity hardware can run cheap inference from very capable, specialized models. Pretty sure it will spawn a new golden age of AI-powered desktop applications. For example, video game space has already been trying…

Whilst I agree with the reservations of the other replies I think you were implying in the future and I'm sure the LLM's will be more trustworthy and up to the task at some point.

What I would really like to see now is all the new TTS models being used more widespread. There are still so many games that have text only output. My kid love Alba: A wildlife Adventure but the eldest still isn't quite ready to read all the text so I have to sit with them reading out all the lines.

If anyone has a way of applying universal mods / accessibility features to existing games I'd love to see someone solve this and happy to help with the TTS!

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#29

Post by chongli »

up2isomorphism wrote: post #12
tomalaci wrote: post #7

With how much NVidia is developing AI-workload accelerating hardware, I expect this will cost maybe few dozen dollars and train in few hours within next few years. What I think will be interesting is when commodity hardware can run cheap inference from very capable, specialized models. Pretty sure it will spawn a new golden age of AI-powered desktop applications. For example, video game space has already been trying…

> For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI). To me this is a downside compared to the NPC generated by humans, since that’s the only reason I would like to read them.

Yeah. I have played a bunch of the roguelike Caves of Qud [1] and it has both hand-written text and procedurally generated text. The former is quite interesting and relevant to both gameplay and plot. The latter is mostly uninteresting and irrelevant, though it does work as "filler." This is similar to how procedurally-generated grass can give a more natural look to a hill than you'd get with tiles (which are incredibly easy to spot unless a ton of work is put into hiding the seams and repeating patterns).

I still long for the day when we can have procedurally-generated stories and quests that are actually interesting to play through. I have no idea how that is going to work though!

[1] https://www.cavesofqud.com

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#30

Post by headcanon »

swatcoder wrote: post #18
tomalaci wrote: post #7

With how much NVidia is developing AI-workload accelerating hardware, I expect this will cost maybe few dozen dollars and train in few hours within next few years. What I think will be interesting is when commodity hardware can run cheap inference from very capable, specialized models. Pretty sure it will spawn a new golden age of AI-powered desktop applications. For example, video game space has already been trying…

> For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI). This'll be a niche for a long, long time. Games are generally carefully crafted to deliver a specific mechanical and/or narrative experience. A world populated by LLM/etc bots or content is one choice of what that experience might be, but it's not going to be a very satisfying one f…

I do think there is a big opportunity for widely supported hardware-accelerated matrix algebra in games. Currently most of that is geared towards graphics (naturally) but being able to easily encode arbitrary models and have them run on-device would open up a lot of opportunities for games (like deep simulation) that weren't possible before. Its currently possible of course but requires custom tooling and (relatively) niche hardware like a high-end graphics card.

I see the development energy around LLMs as a way to open up support for that.

Post reply on HN