Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

231–240 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#231

Post by gitremote »

aurareturn wrote: post #227
gitremote wrote: post #204

Earlier quoted context omitted.

Inference contributes to their losses. In January 2025, Altman admitted they are losing money on Pro subscriptions, because people are using it more than they expected (sending more inference requests per month than would be offset by the monthly revenue). https://xcancel.com/sama/status/1876104315296968813

So people find more value than they thought so they'll just up the price. Meanwhile, they still make more money per inference than they lose.

Currently, they lose more money per inference than they make for Pro subscriptions, because they are essentially renting out their service each month instead of charging for usage (per token).

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#232

Post by hn92726819 »

Invictus0 wrote: post #185
93po wrote: post #155

Earlier quoted context omitted.

they would be break-even if all they did was serve existing models and got rid of everything related to R&D

An AI lab with no R&D. Truly a hacker news moment

I think you maybe have misunderstood the parent (or maybe I did?). They're saying you can't compare an individual's cost to run a model against OpenAI's cost to run it + R&D. Individuals aren't paying for R&D, and that's where most of the cost is.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#233

Post by crawfordcomeaux »

virgil_disgr4ce wrote: post #163
catigula wrote: post #148

Earlier quoted context omitted.

A lot of really smart people working on problems that don't even really need to be solved is an interesting aspect of market allocation.

> working on problems that don't even really need to be solved Very, very few problems _need_ to be solved. Feeding yourself is a problem that needs to be solved in order for you to continue living. People solve problems for different reasons. If you don't think LLMs are valuable, you can just say that.

The few problems humanity has that need to be solved:

1. How to identify humanity's needs on all levels, including cosmic ones...(we're in the Space Age so we need to prepare ourselves for meeting beings from other places)

2. How to meet all of humanity's needs

Pointing this out regularly is probably necessary because the issue isn't why people are choosing what they're doing...it's that our systems actively disincentivize collectibely addressing these two problems in a way that doesn't sacrifice people's wellbeing/lives... and most people don't even think about it like this.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#234

Post by superasn »

nodja wrote: post #222
fergal_reid wrote: post #19

I think the most direct answer is that at scale, inference can be batched, so that processing many queries together in a parallel batch is more efficient than interactively dedicating a single GPU per user (like your home setup). If you want a survey of intermediate level engineering tricks, this post we wrote on the Fin AI blog might be interesting. (There's probably a level of proprietary techniques OpenAI etc have…

This is the real answer, I don't know what people above are even discussing when batching is the biggest reduction in costs. If it costs say $50k to serve one request, with batching is also costs $50k to serve 100 at the same time with minimal performance loss, I don't know what the real number of users is before you need to buy new hardware, but I know it's in the hundreds so going from $50000 to $500 in effective c…

Thanks for the helpful reply! As I wasn't able to fully understand it still, I pasted your reply in chatgpt and asked it some follow up questions and here is what i understand from my interaction:

- Big models like GPT-4 are split across many GPUs (sharding).

- Each GPU holds some layers in VRAM.

- To process a request, weights for a layer must be loaded from VRAM into the GPU's tiny on-chip cache before doing the math.

- Loading into cache is slow, the ops are fast though.

- Without batching: load layer > compute user1 > load again > compute user2.

- With batching: load layer once > compute for all users > send to gpu 2 etc

- This makes cost per user drop massively if you have enough simultaneous users.

- But bigger batches need more GPU memory for activations, so there's a max size.

This does makes sense to me but does this sound accurate to you?

Would love to know if I'm still missing something important.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#235

Post by philwelch »

What incentive do any of the big LLM providers have to solve this problem? I know there are technical reasons, but SaaS is a lucrative and proven business model and the systems have for years all been built by companies with an incentive to keep that model running, which means taking any possible chance to trade off against the possibility of some paying customer ever actually being able to run the software on their own computer. Just like the phone company used to never let you buy a telephone (you had to rent it from the phone company, which is why all the classic Western Electric telephones were indestructible chunks of steel).

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#236

Post by benreesman »

canyon289 wrote: post #6

I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast. https://jax-ml.git…

I don't think it's either useful or particularly accurate to characterize modern disagg racks of inference gear, well-understood RDMA and other low-overhead networking techniques, aggressive MLA and related cache optimizations that are in the literature, and all the other stuff that goes into a system like this as being some kind of mystical thing attended to by a priesthood of people from a different tier of hacker.

This stuff is well understood in public, and where a big name has something highly custom going on? Often as not it's a liability around attachment to some legacy thing. You run this stuff at scale by having the correct institutions and processes in place that it takes to run any big non-trivial system: that's everything from procurement and SRE training to the RTL on the new TPU, and all of the stuff is interesting, but if anyone was 10x out in front of everyone else? You'd be able to tell.

Signed, Someone Who Also Did Megascale Inference for a TOP-5 For a Decade.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#237

Post by knowitnone2 »

93po wrote: post #155
abathologist wrote: post #63

One clever ingredient in OpenAI's secret sauce is billions of dollars of losses. About $5 billion dollars lost in 2024. https://www.cnbc.com/2024/09/27/openai-sees-5-billion-loss-t...

they would be break-even if all they did was serve existing models and got rid of everything related to R&D

they are not the only player so getting rid of R&D would be suicide

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#238

Post by ezst »

abletonlive wrote: post #161
YossarianFrPrez wrote: post #154

Earlier quoted context omitted.

Can you explain what you mean about 'not needing to be solved'? There are versions of that kind of critique that would seem, at least on the surface, to better apply to finance or flash trading. I ask because scaling an system that a substantially chunk of the population finds incredibly useful, including for the more efficient production of public goods (scientific research, for example) does seem like a problem tha…

They won’t be honest and explain it to you but I will. Takes like the one you’re responding to are from loathsome pessimistic anti-llm people that are so far detached from reality they can just confidently assert things that have no bearing on truth or evidence. It’s a coping mechanism and it’s basically a prolific mental illness at this point

And what does that make you? A "loathsome clueless pro-llm zealot detached from reality"? LLMs are essentially next word predictors marketed as oracles. And people use them as that. And that's killing them. Because LLMs don't actually "know", they don't "know that they don't know", and won't tell you they are inadequate when they are. And that's a problem left completely unsolved. At the core of very legitimate concerns about the proliferation of LLMs. If someone here sounds irrational and "coping", it very much appears to be you.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#239

Post by addaon »

computomatic wrote: post #173
0cf8612b2e1e wrote: post #56

Earlier quoted context omitted.

During times of high utilization, how do they handle more requests than they have hardware? Is the software granular enough that they can round robin the hardware per token generated? UserA token, then UserB, then UserC, back to UserA? Or is it more likely that everyone goes into a big FIFO processing the entire request before switching to the next user? I assume the former has massive overhead, but maybe it is worth…

This is great product design at its finest. First of all, they never “handle more requests than they have hardware.” That’s impossible (at least as I’m reading it). The vast majority of usage is via their web app (and free accounts, at that). The web app defaults to “auto” selecting a model. The algorithm for that selection is hidden information. As load peaks, they can divert requests to different levels of hardware…

> But this seems like the easiest one to implement.

Even easier: Just fail. In my experience the ChatGPT web page fails to display (request? generate?) a response between 5% and 10% of the time, depending on time of day. Too busy? Just ignore your customers. They’ll probably come back and try again, and if not, well, you’re billing them monthly regardless.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#240

Post by cootsnuck »

eitally wrote: post #115
airhangerf15 wrote: post #11

An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper" I don't think people realize the size of these compute u…

What I wonder is what this means for Coreweave, Lambda and the rest, who are essentially just renting out fleets of racks like this. Does it ultimately result in acquisition by a larger player? Severe loss of demand? Can they even sell enough to cover the capex costs?

It means they're likely going to be left holding a very expensive bag.
Post reply on HN