Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

321–330 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#321

Post by scarface_74 »

thechao wrote: post #311
rlupi wrote: post #292

Earlier quoted context omitted.

> It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP. > But then again even there, their reputation for abandoning products What are the chances of abandoning TPU-related projects where the company literally invested billions in infrastructure? Zero.

I don't know what scale of "billions" you're talking about; but, Intel blew 1–2 billion on Larrabee. Even worse: Intel blew 5+ billion on mobile pre-iPhone. I remember when that team was shown the door — that's when we had to evaluate the early RGX GPUs as a backstop to try to win Apple's business; the RGX's were turds . Penny-wise pound-foolish.

Intel also wasted untold billions trying to compete with Qualcomm building cellular chips with lackluster results and the sold the division to Apple which has spent billions more just to end up with the lackluster C1 in the SE.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#322

Post by guenthert »

xtiansimon wrote: post #313
semi-extrinsic wrote: post #38

Earlier quoted context omitted.

What do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.

> “They are heavy, noisy like you would not believe, … produces … waste heat.” Haha. I bought a 20 yro IBM server off eBay for a song. It was fun for a minute. Soon became a doorstop and I sold it as pickup-only on eBay for $20. Beast. Never again have one in my home.

Get an AS400. Those were actually expected to be installed in an office, rather than a server room. Might still be perceived as loud at home, but won't be deafening and probably not louder than some gaming rigs.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#323

Post by gitremote »

aurareturn wrote: post #286
gitremote wrote: post #231

Earlier quoted context omitted.

Currently, they lose more money per inference than they make for Pro subscriptions, because they are essentially renting out their service each month instead of charging for usage (per token).

Do you have a source for that?

When an end user asks ChatGPT a question, the chatbot application sends the system prompt, user prompt, and context as input tokens to an inference API, and the LLM generates output tokens for the inference API response.

GPT API inference cost (for developers) is per token (sum of input tokens, cached input tokens, and output tokens per 1M used).

https://openai.com/api/pricing/

https://azure.microsoft.com/en-us/pricing/details/cognitive-...

(Inference cost is charged per token even for free models like Meta LLaMa and DeepSeek-R1 on Amazon Bedrock. https://aws.amazon.com/bedrock/pricing/ )

ChatGPT Pro subscription pricing (the chatbot for end users) is $200/month

https://openai.com/chatgpt/pricing/

"insane thing: we are currently losing money on openai pro subscriptions!

people use it much more than we expected."

- Sam Altman, January 6, 2025

https://xcancel.com/sama/status/1876104315296968813

Again, this means that the average ChatGPT Pro end user's chattiness cost OpenAI too much inference (too many input and output tokens sent and received, respectively, for inference) per month than would be balanced out by OpenAI receiving $200/month in revenue from the average Pro user.

The analogy is like Netflix losing money on their subscriptions because their users watch too much streaming, so they ban account sharing, causing many users to cancel their subscriptions, but this actually helps them become profitable, because the extra users using their service too much generated more costs than revenue.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#324

Post by vikramkr »

0cf8612b2e1e wrote: post #56
cornholio wrote: post #43

Earlier quoted context omitted.

You can knock off a zero or two just by time shifting the 700 million distinct users across a day/week and account for the mere minutes of compute time they will actually use in each interaction. So they might no see peaks higher than 10 million active inference session at the same time. Conversely, you can't do the same thing as a self hosted user, you can't really bank your idle compute for a week and consume it al…

During times of high utilization, how do they handle more requests than they have hardware? Is the software granular enough that they can round robin the hardware per token generated? UserA token, then UserB, then UserC, back to UserA? Or is it more likely that everyone goes into a big FIFO processing the entire request before switching to the next user? I assume the former has massive overhead, but maybe it is worth…

In addition to stuff like that they also handle it with rate limits, that message that Claude would throw almost all the time when they were like "demand is high so you have automatically switched to concise mode", making batch inference cheaper for API customers to convince them to use that instead of real time replies. The site erroring out during a period of high demand also works, prioritizing business customers during a rollout, the service degrading. It's not like any provider has a track record for effortlessly keeping responsiveness super high. Usually it's more the opposite.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#325

Post by thecommakozzi »

brikym wrote: post #179
airhangerf15 wrote: post #11

An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper" I don't think people realize the size of these compute u…

As a non-American the 240V thing made me laugh.

[dead]

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#326

Post by mlyle »

tgma wrote: post #251
mlyle wrote: post #228

Earlier quoted context omitted.

Sure, it's "equally efficient" if you ignore the inefficient thing that is done outside where you draw the system box, directly in proportion to how much you do it. Heating my house with a giant diesel-powered radiant heater from across the street is infinitely efficient, too, since I use no power in my house.

If you don’t close the box of the system at some point to isolate the input, efficiency would be meaningless. I think in the context of the original post, suggesting running a server in winter would be a zero-waste endeavor if you need the heat anyway, it is perfectly clear that the input is electricity to your home at a certain $/kWh and gas at a certain $/BTU. Under that premise, it is fair to say that would not be…

I think this is pretty silly either way.

- There's an upstream loss on electricity directly in proportion to how much you use; ignoring this tilts the analysis in favor of electricity.

- You pay more for heat from electricity than gas, in part because of this loss.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#327

Post by mlyle »

windexh8er wrote: post #316
mlyle wrote: post #226

Earlier quoted context omitted.

> If the 700M users per week is legitimate then my question to you is: how many of those invocations are worth the cost of resources that are spent, in the name of things that are truly productive? Is everything you spend resources on truly productive? Who determines whether something is worth it? Is price/willingness of both parties to transact not an important factor? I don't think ChatGPT can do most things I do.…

I don't believe everything in my world is as efficient as it could be. But I genuinely think about the costs involved [0]. When doing automations that are perfectly handled by deterministic systems why would I put the outcomes of those in the hands of a non-deterministic one? And at that cost differential? We know a few things: LLMs are not efficient, LLMs are consuming more water than traditional compute, we know th…

> When doing automations that are perfectly handled by deterministic systems why would I put the outcomes of those in the hands of a non-deterministic one?

The stuff I'm punting isn't stuff I can automate. It's stuff like, "build me a quick command line tool to model passes from this set of possible orbits" or "convert this bulleted list to a course articulation in the format preferred by the University of California" or "Tell me the 5 worst sentences in this draft and give me proposed fixes."

Human assistants that I would punt this stuff to also consume a lot of wattage and power. ;)

> We didn't have these tools 5 years ago. 5 years ago you dealt with said "drudgery". On the other hand you then say it can't do "most things I do".

I'm not sure why you think this is paradoxical.

I probably eliminate 20-30% of tasks at this point with AI. Honestly, it probably does these tasks better than I would (not better than I could, but you can't give maximum effort on everything). As a result, I get 30-40% more done, and a bigger proportion of it is higher value work.

And, AI sometimes helps me with stuff that I -can't- do, like making a good illustration of something. It doesn't surpass top humans at this stuff, but it surpasses me and probably even where I can get to with reasonable effort.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#329

Post by [deleted] »

frabcus wrote: post #255
superasn wrote: post #234

Earlier quoted context omitted.

Thanks for the helpful reply! As I wasn't able to fully understand it still, I pasted your reply in chatgpt and asked it some follow up questions and here is what i understand from my interaction: - Big models like GPT-4 are split across many GPUs (sharding). - Each GPU holds some layers in VRAM. - To process a request, weights for a layer must be loaded from VRAM into the GPU's tiny on-chip cache before doing the ma…

This seems a bit complicated to me. They don't serve very many models. My assumption is they just dedicate GPUs to specific models, so the model is always in VRAM. No loading per request - it takes a while to load a model in anyway. The limiting factor compared to local is dedicated VRAM - if you dedicate 80GB of VRAM locally 24 hours/day so response times are fast, you're wasting most of the time when you're not que…

[deleted]

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#330

Post by Tade0 »

_zoltan_ wrote: post #170
Tade0 wrote: post #159

Earlier quoted context omitted.

I'm in the market for an oven right now and 230V/16A is the voltage/current the one I'll probably be getting operates under. At 90°C you can do sous vide, so basically use that waste heat entirely. For such temperatures you'd need a CO2 heat pump, which is still expensive. I don't know about gas, as I don't even have a line to my place.

90C for sous vide??? You're going to kill any meal at 90.

Make it "up to 90°C". 5th quarter meats are better done in the higher end of sous vide temperatures.

Point being, you can throttle your equipment to the desired temperature and use that energy effectively.

Post reply on HN