Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#81

Post by quickthrowman »

Dylan16807 wrote: post #64
quickthrowman wrote: post #57

Earlier quoted context omitted.

You’ll need (2) 240V 20A 2P breakers, one for the server and one for the 1-ton mini-split to remove the heat ;)

Matching AC would only need 1/4 the power, right? If you don't already have a method to remove heat.

Cooling BTUs already take the coefficient of performance of the vapor-compression cycle into account. 4w of heat removed for each 1w of input power is around the max COP for an air cooled condenser, but adding an evaporative cooling tower can raise that up to ~7.

I just looked at a spec sheet for a 230V single-phase 12k BTU mini-split and the minimum circuit ampacity was 3A for the air handler and 12A for the condenser, add those together for 15A, divide by .8 is 18.75A, next size up is 20A. Minimum circuit ampacity is a formula that is (roughly) the sum of the full load amps of the motor(s) inside the piece of equipment times 1.25 to determine the conductor size required to power the equipment.

So the condensing unit likely draws ~9.5-10A max and the air handler around ~2.4A, and both will have variable speed motors that would probably only need about half of that to remove 12k BTU of heat, so ~5-6A or thereabouts should do it, which is around 1/3rd of the 16A server, or a COP of 3.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#82

Post by kelnos »

quickthrowman wrote: post #57
semi-extrinsic wrote: post #38

Earlier quoted context omitted.

What do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.

You’ll need (2) 240V 20A 2P breakers, one for the server and one for the 1-ton mini-split to remove the heat ;)

Well, get a heat pump with a good COP of 3 or more, and you won't need quite as much power ;)

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#84

Post by dortlick »

mattmanser wrote: post #62
invaliduser wrote: post #22

Earlier quoted context omitted.

Even is the AI bubble does not pops, your prediction about those servers being available on ebay in 10 years will likely be true, because some datacenters will simply upgrade their hardware and resell their old ones to third parties.

Someone's take on AI was that we're collectively investing billions in data centers that will be utterly worthless in 10 years. Unlike the investments in railways or telephone cables or roads or any other sort of architecture, this investment has a very short lifespan. Their point was that whatever your take on AI, the present investment in data centres is a ridiculous waste and will always end up as a huge net loss…

Sure, but what about the collective investment in smartphones, digital cameras, laptops, even cars. Not much modern technology is useful and practical after 10 years, let alone 20. AI is probably moving a little faster than normal, but technology depreciation is not limited to AI.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#85

Post by ignoramous »

canyon289 wrote: post #6

I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast. https://jax-ml.git…

> Another great resource to look at is the unsloth guides.

And folks at LMSys: https://lmsys.org/blog/

  Large Model Systems (LMSYS Corp.) is a 501(c)(3) non-profit focused on incubating open-source projects and research. Our mission is to make large AI models accessible to everyone by co-developing open models, datasets, systems, and evaluation tools. We conduct cutting-edge machine learning research, develop open-source software, train large language models for broad accessibility, and build distributed systems to optimize their training and inference.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#86

Post by ritz_labringue »

The short answer is "batch size". These days, LLMs are what we call "Mixture of Experts", meaning they only activate a small subset of their weights at a time. This makes them a lot more efficient to run at high batch size.

If you try to run GPT4 at home, you'll still need enough VRAM to load the entire model, which means you'll need several H100s (each one costs like $40k). But you will be under-utilizing those cards by a huge amount for personal use.

It's a bit like saying "How come Apple can make iphones for billions of people but I can't even build a single one in my garage"

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#87

Post by Scoundreller »

quickthrowman wrote: post #57
semi-extrinsic wrote: post #38

Earlier quoted context omitted.

What do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.

You’ll need (2) 240V 20A 2P breakers, one for the server and one for the 1-ton mini-split to remove the heat ;)

Just air freight them from 60 degrees North to 60 degrees South and vice verse every 6 months.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#88

Post by potatolicious »

torginus wrote: post #71
airhangerf15 wrote: post #11

An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper" I don't think people realize the size of these compute u…

I wonder if it's feasible to hook up NAND flash with a high bandwidth link necessary for inference. Each of these NAND chips hundreds of dies of flash stacked inside, and they are hooked up to the same data line, so just 1 of them can talk at the same time, and they still achieve >1GB/s bandwidth. If you could hook them up in parallel, you could have 100s of GBs of bandwidth per chip.

NAND is very, very slow relative to RAM, so you'd pay a huge performance penalty there. But maybe more importantly my impression is that memory contents mutate pretty heavily during inference (you're not just storing the fixed weights), so I'd be pretty concerned about NAND wear. Mutating a single bit on a NAND chip a million times over just results in a large pile of dead NAND chips.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#89

Post by [deleted] »

kelnos wrote: post #82
quickthrowman wrote: post #57

Earlier quoted context omitted.

You’ll need (2) 240V 20A 2P breakers, one for the server and one for the 1-ton mini-split to remove the heat ;)

Well, get a heat pump with a good COP of 3 or more, and you won't need quite as much power ;)

[deleted]

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#90

Post by ksherlock »

semi-extrinsic wrote: post #38
airhangerf15 wrote: post #11

An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper" I don't think people realize the size of these compute u…

What do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.

It's not waste heat if you only run it in the winter.
Post reply on HN