Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#141

Post by handfuloflight »

michaelt wrote: post #52
neko_ranger wrote: post #21

Earlier quoted context omitted.

Four H100 in a 2U rack didn't sound impressive, but that is accurate: >A typical 1U or 2U server can accommodate 2-4 H100 PCIe GPUs, depending on the chassis design. >In a 42U rack with 20x 2U servers (allowing space for switches and PDU), you could fit approximately 40-80 H100 PCIe GPUs.

Why stop at 80 H100s for a mere 6.4 terabytes of GPU memory? Supermicro will sell you a full rack loaded with servers [1] providing 13.4 TB of GPU memory. And with 132kW of power output, you can heat an olympic-sized swimming pool by 1°C every day with that rack alone. That's almost as much power consumption as 10 mid-sized cars cruising at 50 mph. [1] https://www.supermicro.com/en/products/system/gpu/48u/srs-gb...

What about https://www.cerebras.ai/system?

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#142

Post by afr0ck »

Inference runs like a stateless web server. If you have 50K or 100K machines, each with a tons of GPUs (usually 8 GPUs per node), then you end up with a massive GPU infrastructure that can run hundreds of thousands, if not millions, of inference instances. They use something like Kubernetes on top for scheduling, scaling and spinning up instances as needed.

For storage, they also have massive amount of hard disks and SSD behind planet scale object file systems (like AWS's S3 or Tectonic at Meta or MinIO in prem) all connected by massive amount of switches and routers of varying capacity.

So in the end, it's just the good old Cloud, but also with GPUs.

Btw, OpenAI's infrastructure is provided and managed by Microsoft Azure.

And, yes, all of this requires billions of dollars to build and operate.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#143

Post by 7speter »

Elsewhere in the thread, someone talked about how h100’s each have 80GB of vram and cost 20000 dollars.

The largest chatgpt models are maybe 1-1.5tb in size and all of that needs to load into pooled vram. That sounds daunting, but a company like open ai has countless machines that have enough of these datacenter grade gpus with gobs of vram pooled together to run their big models.

Inference is also pretty cheap, especially when a model can comfortably fit in a pool of vram. Its not that the pool of gpus spool up each time someone sends a request, but whats more likely is that there’s a queue to f requests from someone like chatgpts 700 million users, and the multiple (I have no idea how many) pools of vram keep the models in their memory to chew through that nearly perpetual queue of requests.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#144

Post by sixo »

jl6 wrote: post #41

If the explanation really is, as many comments here suggest, that prompts can be run in parallel in batches at low marginal additional cost, then that feels like bad news for the democratization and/or local running of LLMs. If it’s only cost-effective to run a model for ~thousands of people at the same time, it’s never going to be cost-effective to run on your own.

That determines the cost effectiveness to make it worth it to train one of these models in the first place. Using someone else's weights, you can afford to predict quite inefficiently.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#146

Post by nbardy »

The serving infrastructure becomes very efficient when serving requests in parallel.

Look at VLLM. It's the top open source version of this.

But the idea is you can service 5000 or so people in parallel.

You get about 1.5-2x slowdown on per token speed per user, but you get 2000x-3000x throughput on the server.

The main insight is that memory bandwidth is the main bottleneck so if you batch requests and use a clever KV cache along with the batching you can drastically increase parallel throughput.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#147

Post by oceanplexian »

CamperBob2 wrote: post #72
semi-extrinsic wrote: post #38

Earlier quoted context omitted.

What do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.

Are you talking about the guy in Temecula running two different auctions with some of the same photos (356878140643 and 357146508609, both showing a missing heat sink?) Interesting, but seems sketchy. How useful is this Tesla-era hardware on current workloads? If you tried to run the full DeepSeek R1 model on it at (say) 4-bit quantization, any idea what kind of TTFT and TPS figures might be expected?

I can’t speak to the Tesla stuff but I run an Epyc 7713 with a single 3090 and creatively splitting the model between GPU/8 channels of DDR4 I can do about 9 tokens per second on a q4 quant.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#148

Post by catigula »

canyon289 wrote: post #6

I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast. https://jax-ml.git…

A lot of really smart people working on problems that don't even really need to be solved is an interesting aspect of market allocation.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#149

Post by dboreham »

airhangerf15 wrote: post #11

An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper" I don't think people realize the size of these compute u…

They'll be in landfill in 10 years.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#150

Post by cududa »

ritz_labringue wrote: post #86

The short answer is "batch size". These days, LLMs are what we call "Mixture of Experts", meaning they only activate a small subset of their weights at a time. This makes them a lot more efficient to run at high batch size. If you try to run GPT4 at home, you'll still need enough VRAM to load the entire model, which means you'll need several H100s (each one costs like $40k). But you will be under-utilizing those card…

Great metaphor
Post reply on HN