Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

github.com

Post reply on HN
Print view View original

41–50 of 62 posts

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#41

Post by 123yawaworht456 »

swatcoder wrote: post #18
tomalaci wrote: post #7

With how much NVidia is developing AI-workload accelerating hardware, I expect this will cost maybe few dozen dollars and train in few hours within next few years. What I think will be interesting is when commodity hardware can run cheap inference from very capable, specialized models. Pretty sure it will spawn a new golden age of AI-powered desktop applications. For example, video game space has already been trying…

> For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI). This'll be a niche for a long, long time. Games are generally carefully crafted to deliver a specific mechanical and/or narrative experience. A world populated by LLM/etc bots or content is one choice of what that experience might be, but it's not going to be a very satisfying one f…

for narrative/dialogue, yes, text generation is currently useless. censorship, slop, extreme positivity bias. even jailbroken Opus is shit.

but audio generation we already have is pretty much good enough, and this is big. it's not AAA tier yet, sure, but still lightyears better half-assing it with mediocre voice actors. it is now an option to only use real voice actors for a few key characters, and even that won't be necessary within the next decade. even indie video games will be fully-voiced soon.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#42

Post by HPsquared »

tomalaci wrote: post #7

With how much NVidia is developing AI-workload accelerating hardware, I expect this will cost maybe few dozen dollars and train in few hours within next few years. What I think will be interesting is when commodity hardware can run cheap inference from very capable, specialized models. Pretty sure it will spawn a new golden age of AI-powered desktop applications. For example, video game space has already been trying…

Similar scaling to genome sequencing. First genome was a huge undertaking, now routine after a few Moore-esque cycles.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#43

Post by swatcoder »

jetrink wrote: post #39
swatcoder wrote: post #18

Earlier quoted context omitted.

> For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI). This'll be a niche for a long, long time. Games are generally carefully crafted to deliver a specific mechanical and/or narrative experience. A world populated by LLM/etc bots or content is one choice of what that experience might be, but it's not going to be a very satisfying one f…

Those don't have to be mutually exclusive though. To take AI out of it, think of those murder mystery parties where actors interact with attendees. Actors have roles to play and things they must do to move the story forward, but they improvise their dialog when talking with the players and sometimes each other. Or if you've ever played D&D, you have experienced talking with NPCs that are controlled by your DM. I thin…

Yes, that's the intuition for where the technology might go. Someday.

But actors and DM's are much more disciplined than LLM's, partly because they have careers and friendships on the line for misbehavior. For what amazing things they can do in good weather, LLM's are not really reliable when you want them to consistently deliver something very specific, very secure, or very artfully crafted. They may get there, but their design makes it a very hard problem that we're still a long way from seeing commercialized.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#44

Post by Tiberium »

forrestthewoods wrote: post #36
Tiberium wrote: post #16

Earlier quoted context omitted.

Are you sure? Models like Claude 3.5 Sonnet are both good at writing and instructions, as long as you set some guardrails for the model, they can be great NPCs.

> Are you sure? Absolutely. Current LLMs have insufficient world state. Imagine a game like Stardew Valley. It's got a town with 30 NPCs or some such. They all have personalities and the player builds a relationship with them over time. Current LLMs can't do that. They hallucinate waaaaaaay too much. You can't reliably define and evolve relationships. Amongst many other short comings. I'm super pro AI and use ChatGPT…

But you're saying that you want a single model to handle all NPCs and the whole world. Of course this isn't possible currently. But using a separate model with separate context for each character is. Also, if you use ChatGPT for programming, try Claude 3.5 Sonnet - it's really better than GPT-4o for programming.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#45

Post by forrestthewoods »

Tiberium wrote: post #44
forrestthewoods wrote: post #36

Earlier quoted context omitted.

> Are you sure? Absolutely. Current LLMs have insufficient world state. Imagine a game like Stardew Valley. It's got a town with 30 NPCs or some such. They all have personalities and the player builds a relationship with them over time. Current LLMs can't do that. They hallucinate waaaaaaay too much. You can't reliably define and evolve relationships. Amongst many other short comings. I'm super pro AI and use ChatGPT…

But you're saying that you want a single model to handle all NPCs and the whole world. Of course this isn't possible currently. But using a separate model with separate context for each character is. Also, if you use ChatGPT for programming, try Claude 3.5 Sonnet - it's really better than GPT-4o for programming.

> But you're saying that you want a single model to handle all NPCs and the whole world.

No, I did not say that at all. I didn't specify how the LLMs may or may not be structure. I'm saying that current LLMs - and yes I've used Claude 3.5 Sonnet - are insufficient. There is no existence proof that they're sufficient.

LLMs are great. They aren't great enough for video game NPC. Not yet. Further innovation is needed. You're free to disagree. I can't prove a dispositive. But there is no working example.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#46

Post by glial »

123yawaworht456 wrote: post #41
swatcoder wrote: post #18

Earlier quoted context omitted.

> For example, video game space has already been trying to create AI-powered NPCs, world generation and story-telling (e.g. Inworld AI). This'll be a niche for a long, long time. Games are generally carefully crafted to deliver a specific mechanical and/or narrative experience. A world populated by LLM/etc bots or content is one choice of what that experience might be, but it's not going to be a very satisfying one f…

for narrative/dialogue, yes, text generation is currently useless. censorship, slop, extreme positivity bias. even jailbroken Opus is shit. but audio generation we already have is pretty much good enough, and this is big. it's not AAA tier yet, sure, but still lightyears better half-assing it with mediocre voice actors. it is now an option to only use real voice actors for a few key characters, and even that won't be…

Similar for textures - it would be really neat if textures were auto-generated to add detail when you get close to something. As it is, the sprites just look bad.

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#47

Post by jononor »

swatcoder wrote: post #43
jetrink wrote: post #39

Earlier quoted context omitted.

Those don't have to be mutually exclusive though. To take AI out of it, think of those murder mystery parties where actors interact with attendees. Actors have roles to play and things they must do to move the story forward, but they improvise their dialog when talking with the players and sometimes each other. Or if you've ever played D&D, you have experienced talking with NPCs that are controlled by your DM. I thin…

Yes, that's the intuition for where the technology might go. Someday. But actors and DM's are much more disciplined than LLM's, partly because they have careers and friendships on the line for misbehavior. For what amazing things they can do in good weather, LLM's are not really reliable when you want them to consistently deliver something very specific, very secure, or very artfully crafted. They may get there, but…

Agreed that "live" in-game LLMs for NPCs is probably a little while away. Need better tools than now that allows to strike a balance between constrained/directed output and variation. I suspect that LLMs are already used by (some) game developers to aided dialogue creation tools for game developers, where they curate and select more or less a fixed vocabulary. I do think that there are many viable steps to "generic". For example, if the game developer could specify possible dialog trees (as in the possible branches and outcomes), then an LLM could ideally fill in the concrete text when reaching the different paths. Or an LLM could add in "small talk" - meaning things that reflect the world state, recent events, but still have the quest type dialog be practically hardcoded (so there is no risk of unreachable states etc).

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#48

Post by ebalit »

forrestthewoods wrote: post #45
Tiberium wrote: post #44

Earlier quoted context omitted.

But you're saying that you want a single model to handle all NPCs and the whole world. Of course this isn't possible currently. But using a separate model with separate context for each character is. Also, if you use ChatGPT for programming, try Claude 3.5 Sonnet - it's really better than GPT-4o for programming.

> But you're saying that you want a single model to handle all NPCs and the whole world. No, I did not say that at all. I didn't specify how the LLMs may or may not be structure. I'm saying that current LLMs - and yes I've used Claude 3.5 Sonnet - are insufficient. There is no existence proof that they're sufficient. LLMs are great. They aren't great enough for video game NPC. Not yet. Further innovation is needed. Y…

As a gamedev, what would you use currently instead for NPCs? And what would you say makes this/those solutions better than LLMs?

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#49

Post by latchkey »

karpathy wrote: post #32
mmoskal wrote: post #23

Earlier quoted context omitted.

H100 SXM is 2000 TFLOPS at FP16. Multiply by 8. M3 Max is 28 TFLOPS at FP16. Based on FLOPS alone, it would be more like a year or two.

(H100 SXM is 1000 TFLOPS, *2 is from "with sparsity", which is not used here.)

MI300x is 1300 TFLOPs at FP16 (without sparsity). Looking forward to seeing the results.

https://www.amd.com/en/products/accelerators/instinct/mi300/...

Re: Karpathy: Let's reproduce GPT-2 (1.6B): one 8XH100 node 24h $672 in llm.c

#50

Post by forrestthewoods »

ebalit wrote: post #48
forrestthewoods wrote: post #45

Earlier quoted context omitted.

> But you're saying that you want a single model to handle all NPCs and the whole world. No, I did not say that at all. I didn't specify how the LLMs may or may not be structure. I'm saying that current LLMs - and yes I've used Claude 3.5 Sonnet - are insufficient. There is no existence proof that they're sufficient. LLMs are great. They aren't great enough for video game NPC. Not yet. Further innovation is needed. Y…

As a gamedev, what would you use currently instead for NPCs? And what would you say makes this/those solutions better than LLMs?

Dialogue systems vary by game and are generally custom. But they amount to hard coded if-else branches if you squint. The only one I ever wrote was a simple announcer for a sports-ish shooter.

Your question is weird. Every video game ever made has shipped using not LLMs. Not a single commercial game has ever shipped with an LLM. So I’d say what makes classic NPC systems better is they’ve shipped tens of thousands of commercial successes over 50+ years. And what makes LLMs worse is they haven’t once been proven viable for even a single title. Nor have they produced even a compelling tech demo.

Geez people.

Post reply on HN