Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

331–340 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#331

Post by yencabulator »

xtiansimon wrote: post #313
semi-extrinsic wrote: post #38

Earlier quoted context omitted.

What do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.

> “They are heavy, noisy like you would not believe, … produces … waste heat.” Haha. I bought a 20 yro IBM server off eBay for a song. It was fun for a minute. Soon became a doorstop and I sold it as pickup-only on eBay for $20. Beast. Never again have one in my home.

That's about the era my company was an IBM reseller. Once I was kneeling behind 8x1U starting up and all the fans went to max speed for 3 seconds. Never put rackmount hardware in a room that is near anything living.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#334

Post by nwhnwh »

canyon289 wrote: post #6

I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast. https://jax-ml.git…

Nothing smart about making something that is not useful for humans.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#335

Post by KaiserPro »

abdullin wrote: post #310
KaiserPro wrote: post #95

Earlier quoted context omitted.

Same explanation but with less mysticism: Inference is (mostly) stateless. So unlike training where you need to have memory coherence over something like 100k machines and somehow avoid the certainty of machine failure, you just need to route mostly small amounts of data to a bunch of big machines. I don't know what the specs of their inference machines are, but where I worked the machines research used were all 8gpu…

> Inference is (mostly) stateless Quite the opposite. Context caching requires state (K/V cache) close to the VRAM. Streaming requires state. Constrained decoding (known as Structured Outputs) also requires state.

> Quite the opposite.

Unless something has dramatically changed, the model is stateless. The context cache needs to be injected before the new prompt, but for what I understand (and please do correct me if I'm wrong) the the context cache isn't that big, like in the order of a few tens of kilobytes. Plus the cache saves seconds of GPU time, so having an extra 100ms of latency is nothing compare to a cache miss. so a broad cache is much much better than a narrow local cache.

But! even if its larger, Your bottleneck isn't the network, its waiting on the GPUs to be free[1]. So whilst having the cache really close ie in the same rack, or same machine, will give the best performance, it will limit your scale (because the cache is only effective for a small number of users)

[1] a 100megs of data shared over the same datacentre network every 2-3 seconds per node isn't that much, especially if you have a partitioned network (ie like AWS where you have a block network and a "network" network)

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#336

Post by RagnarD »

airhangerf15 wrote: post #11

An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper" I don't think people realize the size of these compute u…

An RTX 6000 Pro (NVIDIA Blackwell GPU) has 96GB of VRAM and can be had for around $7700 currently (at least, the lowest price I've found.) It plugs into standard PC motherboard PCIe slots. The Max Q edition has slightly less performance but a max TDP of only 300W.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#337

Post by joshhart »

bawana wrote: post #320
joshhart wrote: post #250

A single node with GPUs has a lot of FLOPs and very high memory bandwidth. When only processing a few requests at a time, the GPUs are mostly waiting on the model weights to stream from the GPU ram to the processing units. When batching requests together, they can stream a group of weights and score many requests in parallel with that group of weights. That allows them to have great efficiency. Some of the other main…

How is speculative decoding helpful if you still have to run the full model against which you check the results?

So the inference speed at low to medium usage is memory bandwidth bound, not compute bound. By “forecasting” into the future you do not increase the memory bandwidth pressure much but you use more compute. The compute is checking each potential token in parallel for several tokens forward. That compute is essentially free though because it’s not the limiting resource. Hope this makes sense, tried to keep it simple.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#338

Post by ranger_danger »

LAC-Tech wrote: post #193
canyon289 wrote: post #6

I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast. https://jax-ml.git…

If people at google are so smart why can't google.com get a 100% lighthouse score?

Pro-tip they're just not. A lot of tech nerds really like to think they're a genius with all the answers ("why don't they just do XX"), but some eventually learn that the world is not so black and white.

The Dunning-Kruger effect also applies to smart people. You don't stop when you are estimating your ability correctly. As you learn more, you gain more awareness of your ignorance and continue being conservative with your self estimates.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#339

Post by jeffrallen »

patrick451 wrote: post #257
zozbot234 wrote: post #248

Earlier quoted context omitted.

> Inference is (mostly) stateless. ... you just need to route mostly small amounts of data to a bunch of big machines. I think this might just be the key insight. The key advantage of doing batched inference at a huge scale is that once you maximize parallelism and sharding, your model parameters and the memory bandwidth associated with them are essentially free (since at any given moment they're being shared among a…

> The key advantage of doing batched inference at a huge scale is that once you maximize parallelism and sharding, your model parameters and the memory bandwidth associated with them are essentially free (since at any given moment they're being shared among a huge amount of requests!) Kind of unrelated, but this comment made me wonder when we will start seeing side channel attacks that force queries to leak into each…

I asked a colleague about this recently and he explained it away with a wave of the hand saying, "different streams of tokens and their context are on different ranks of the matrices". And I kinda believed him, based on the diagrams I see on Welch Labs YouTube channel.

On the other hand, I've learned that when I ask questions about security to experts in a field (who are not experts in security) I almost always get convincing hand waves, and they are almost always proven to be completely wrong.

Sigh.

Post reply on HN