Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

271–280 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#271

Post by procaryote »

michaelt wrote: post #52
neko_ranger wrote: post #21

Earlier quoted context omitted.

Four H100 in a 2U rack didn't sound impressive, but that is accurate: >A typical 1U or 2U server can accommodate 2-4 H100 PCIe GPUs, depending on the chassis design. >In a 42U rack with 20x 2U servers (allowing space for switches and PDU), you could fit approximately 40-80 H100 PCIe GPUs.

Why stop at 80 H100s for a mere 6.4 terabytes of GPU memory? Supermicro will sell you a full rack loaded with servers [1] providing 13.4 TB of GPU memory. And with 132kW of power output, you can heat an olympic-sized swimming pool by 1°C every day with that rack alone. That's almost as much power consumption as 10 mid-sized cars cruising at 50 mph. [1] https://www.supermicro.com/en/products/system/gpu/48u/srs-gb...

> as much power consumption as 10 mid-sized cars cruising at 50 mph

Imperial units are so weird

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#272

Post by lhl »

vrm wrote: post #42
tough wrote: post #20

Earlier quoted context omitted.

gpt-oss-120b can be used with gpt-oss-20b as speculative drafting on LM Studio I'm not sure it improved the speed much

a 6:1 parameter ratio is too small for specdec to have that much of an effect. You'd really want to see 10:1 or even more for this to start to matter

You're right on ratios, but actually the ratio is much worse than 6:1 since they are MoEs. The 20B has 3.6B active, and the 120B has only 5.1B active, only about 40% more!

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#273

Post by aniviacat »

nocturnes wrote: post #258
addaon wrote: post #239

Earlier quoted context omitted.

> But this seems like the easiest one to implement. Even easier: Just fail. In my experience the ChatGPT web page fails to display (request? generate?) a response between 5% and 10% of the time, depending on time of day. Too busy? Just ignore your customers. They’ll probably come back and try again, and if not, well, you’re billing them monthly regardless.

Is this a common experience for others? In several years of reasonable ChatGPT use I have only experienced that kind of failure a couple of times.

I don't usually see responses fail. But what I did see shortly after the GPT-5 release (when servers were likely overloaded) was the model "thinking" for over 8 minutes. It seems like (if you manually select the model) you're simply getting throttled (or put in a queue).

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#274

Post by DecentShoes »

invaliduser wrote: post #22
airhangerf15 wrote: post #11

An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper" I don't think people realize the size of these compute u…

Even is the AI bubble does not pops, your prediction about those servers being available on ebay in 10 years will likely be true, because some datacenters will simply upgrade their hardware and resell their old ones to third parties.

This seems likely. Blizzard even sold off old World of Warcraft servers. You can still get them on ebay

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#275

Post by big_hacker »

bee_rider wrote: post #209
JoshuaDavid wrote: post #130

Earlier quoted context omitted.

Google employees collectively have a lot of talent.

A truly astonishing amount of talent applied to… hosting emails very well, and losing the search battle against SEO spammers.

Well, Search had no chance when the sites also make money from Google ads. Google fucked their Search by creating themselves incentives for bounce rate.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#276

Post by gregoriol »

abathologist wrote: post #63

One clever ingredient in OpenAI's secret sauce is billions of dollars of losses. About $5 billion dollars lost in 2024. https://www.cnbc.com/2024/09/27/openai-sees-5-billion-loss-t...

With infinite resources, you can serve infinite users. Until it's gone.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#277

Post by oblio »

aurareturn wrote: post #227
gitremote wrote: post #204

Earlier quoted context omitted.

Inference contributes to their losses. In January 2025, Altman admitted they are losing money on Pro subscriptions, because people are using it more than they expected (sending more inference requests per month than would be offset by the monthly revenue). https://xcancel.com/sama/status/1876104315296968813

So people find more value than they thought so they'll just up the price. Meanwhile, they still make more money per inference than they lose.

This assumes that the value obtained by customers is high enough to cover any possible actual cost.

Many current AI uses are low value things or one time things (for example CV generation, which is killing online hiring).

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#278

Post by lhl »

jl6 wrote: post #41

If the explanation really is, as many comments here suggest, that prompts can be run in parallel in batches at low marginal additional cost, then that feels like bad news for the democratization and/or local running of LLMs. If it’s only cost-effective to run a model for ~thousands of people at the same time, it’s never going to be cost-effective to run on your own.

For fungible things, it's easy to cost out. But not all things can be broken down just in token cost, especially as people start building their lives around specific models.

Even beyond privacy just the availability is out of your control - you can look at r/ChatGPT's collective spasm yesterday when 4o was taken from them, but basically, you have no guarantees to access for services, and for LLM models in particular, "upgrades" can completely change behavior/services that you depend on.

Google has been even worse in the past here, I've seen them deprecate model versions with 1 month notices. It seems a lot of model providers are doing dynamic model switching/quanting/reasoning effort adjustments based on load now.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#279

Post by torginus »

slickytail wrote: post #128
torginus wrote: post #101

Earlier quoted context omitted.

No it's not slow - a single NAND chip in SSDs offers >1GB of bandwidth - inside the chip there are 100+ wafers actually holding the data, but in SSDs only one of them is active when reading/writing. You could probably make special NAND chips where all of them can be active at the same time, which means you could get 100GB+ bandwidth out of a single chip. This would be useless for data storage scenarios, but very usef…

The memory bandwidth on an H100 is 3TB/s, for reference. This number is the limiting factor in the size of modern LLMs. 100GB/s isn't even in the realm of viability.

That bandwidth is for the whole GPU, which has 6 chips. But anyways, what I'm proposing isn't for the high-end and training, but for making inference cheap.

And I was somehat conservative with the numbers, a modern budget SSD with a single NAND can do more than 5GB/s read speed.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#280

Post by torginus »

slickytail wrote: post #128
torginus wrote: post #101

Earlier quoted context omitted.

No it's not slow - a single NAND chip in SSDs offers >1GB of bandwidth - inside the chip there are 100+ wafers actually holding the data, but in SSDs only one of them is active when reading/writing. You could probably make special NAND chips where all of them can be active at the same time, which means you could get 100GB+ bandwidth out of a single chip. This would be useless for data storage scenarios, but very usef…

The memory bandwidth on an H100 is 3TB/s, for reference. This number is the limiting factor in the size of modern LLMs. 100GB/s isn't even in the realm of viability.

That bandwidth is for the whole GPU, which has 6 mermoy chips. But anyways, what I'm proposing isn't for the high-end and training, but for making inference cheap.

And I was somehat conservative with the numbers, a modern budget SSD with a single NAND can do more than 5GB/s read speed.

Post reply on HN