Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

github.com

Post reply on HN
Print view View original

31–40 of 62 posts

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#31

Post by charlescurt123 »

tomalaci wrote: post #7

With how much NVidia is developing AI-workload accelerating hardware, I expect this will cost maybe few dozen dollars and train in few hours within next few years. What I think will be interesting is when commodity hardware can run cheap inference from very capable, specialized models. Pretty sure it will spawn a new golden age of AI-powered desktop applications. For example, video game space has already been trying…

I imagine we could do this now but not the way you think.

have a human created story and text as a guideline.

With that have genAI make the text per stage, you would get different statements every time and would stay on track.

Would be interesting to play a game where all players say the same information in slightly different ways every single playthrough.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#32

Post by karpathy »

mmoskal wrote: post #23
jamestimmins wrote: post #15

Earlier quoted context omitted.

Ah so I couldn't just run this on my laptop for ~48 hours? That's too bad.

H100 SXM is 2000 TFLOPS at FP16. Multiply by 8. M3 Max is 28 TFLOPS at FP16. Based on FLOPS alone, it would be more like a year or two.

(H100 SXM is 1000 TFLOPS, *2 is from "with sparsity", which is not used here.)

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#33

Post by swatcoder »

LeanderK wrote: post #21
MisterBastahrd wrote: post #17

Earlier quoted context omitted.

You don't even necessarily need to have them coming up with valid speech. Simply giving out random quests and rewards would keep people running on a loot treadmill for most open world multiplayer games.

I think at first even background NPCs that don't give quests and rewards would be nice. Sort of give everyone a character and let them just babble. It breaks the immersion to hear the same phrases repeatedly. You can still handcraft every important quest and interaction to achieve high fidelity, but I would like random NPCs you bump into to not just repeat things all the time.

You're more likely to see studios (who want lots of more varied content) use generative AI in the studio, where they might generate and review it before release.

Letting generators run free on the client sets up different kinds of immersion-breaking, where NPC's hallucinate misleading details about the story/world, can be tricked into reciting off-topic absurdities or age-rating violations, etc. AAA studios can't afford the embarrassment of that and smaller designers with pride of craft won't see their signature come through the art in it. Surely, some designers will figure out ways to make it work great for some specific idea, but it's not the best way to use the technology in most cases.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#35

Post by rty32 »

michaelmior wrote: post #9
jamestimmins wrote: post #4

Anyone have an idea if this is feasible to do on a Macbook with a built-in GPU?

Probably not with the same amount of training time, but I'd imagine a recent MBP GPU could handle GPT-2 training. The biggest challenge is that the training would need to be reimplemented for Metal instead of CUDA.

Slightly off topic -- I just saw people saying how Mac's unified memory makes it a strength to train models on Macs: https://www.macrumors.com/2024/07/10/apple-leads-global-pc-g..., and how energy efficient they are etc. But what I am seeing is that people don't often even touch Macs at all -- they write code with CUDA and that's it. I find this kind of conversation fascinating.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#36

Post by forrestthewoods »

Tiberium wrote: post #16
forrestthewoods wrote: post #13

Earlier quoted context omitted.

> For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI). Current AI isn't even close to good enough for video game NPCs and related. We're several breakthroughs away from that being possible at any cost. Those breakthroughs might happen in 3 years, or they might not happen in 10. Hard to predict.

Are you sure? Models like Claude 3.5 Sonnet are both good at writing and instructions, as long as you set some guardrails for the model, they can be great NPCs.

> Are you sure?

Absolutely.

Current LLMs have insufficient world state. Imagine a game like Stardew Valley. It's got a town with 30 NPCs or some such. They all have personalities and the player builds a relationship with them over time. Current LLMs can't do that. They hallucinate waaaaaaay too much. You can't reliably define and evolve relationships. Amongst many other short comings.

I'm super pro AI and use ChatGPT all the time for programming. So I'm not being an AI hater. But I am a gamedev and I can say that what exists today simply isn't good enough.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#37

Post by techjamie »

MisterBastahrd wrote: post #17
up2isomorphism wrote: post #12

Earlier quoted context omitted.

> For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI). To me this is a downside compared to the NPC generated by humans, since that’s the only reason I would like to read them.

You don't even necessarily need to have them coming up with valid speech. Simply giving out random quests and rewards would keep people running on a loot treadmill for most open world multiplayer games.

You could maybe pull this off in a game like Borderlands where the loot is basically just the same dozen guns but with different numbers and effects. But as is, the LLM text isn't going to be much different than a sufficiently large AdLib system.

I think there is value to be gained in having LLMs as part of the development process, maybe even the game itself, but I think conventional methods are about as sufficient for quests.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#38

Post by mmoskal »

karpathy wrote: post #32
mmoskal wrote: post #23

Earlier quoted context omitted.

H100 SXM is 2000 TFLOPS at FP16. Multiply by 8. M3 Max is 28 TFLOPS at FP16. Based on FLOPS alone, it would be more like a year or two.

(H100 SXM is 1000 TFLOPS, *2 is from "with sparsity", which is not used here.)

Right... and there are probably some communication overheads over NVLink that would not be present on single laptop. So a few months maybe :)

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#39

Post by jetrink »

swatcoder wrote: post #18
tomalaci wrote: post #7

With how much NVidia is developing AI-workload accelerating hardware, I expect this will cost maybe few dozen dollars and train in few hours within next few years. What I think will be interesting is when commodity hardware can run cheap inference from very capable, specialized models. Pretty sure it will spawn a new golden age of AI-powered desktop applications. For example, video game space has already been trying…

> For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI). This'll be a niche for a long, long time. Games are generally carefully crafted to deliver a specific mechanical and/or narrative experience. A world populated by LLM/etc bots or content is one choice of what that experience might be, but it's not going to be a very satisfying one f…

Those don't have to be mutually exclusive though. To take AI out of it, think of those murder mystery parties where actors interact with attendees. Actors have roles to play and things they must do to move the story forward, but they improvise their dialog when talking with the players and sometimes each other. Or if you've ever played D&D, you have experienced talking with NPCs that are controlled by your DM. I think video game AI could be a lot like that, where NPCs use natural language instead of rigid dialog trees, but otherwise, they behave a lot like they do today.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#40

Post by thatguymike »

jetrink wrote: post #39
swatcoder wrote: post #18

Earlier quoted context omitted.

> For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI). This'll be a niche for a long, long time. Games are generally carefully crafted to deliver a specific mechanical and/or narrative experience. A world populated by LLM/etc bots or content is one choice of what that experience might be, but it's not going to be a very satisfying one f…

Those don't have to be mutually exclusive though. To take AI out of it, think of those murder mystery parties where actors interact with attendees. Actors have roles to play and things they must do to move the story forward, but they improvise their dialog when talking with the players and sometimes each other. Or if you've ever played D&D, you have experienced talking with NPCs that are controlled by your DM. I thin…

Ah, like this: https://www.youtube.com/watch?v=Kw51fkRiKZU
Post reply on HN