Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

171–180 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#171
post #124

Earlier quoted context omitted.

Heat pump sure, but how is gas furnace more efficient than resistive load inside the house? Do you mean more economical rather than more efficient (due to gas being much cheaper/unit of energy)?

Depends where your electricity comes from. If you're burning fossil fuels to make electricity, that's only about 40% efficient, so you need to burn 2.5x as much fuel to get the same amount of heat into the house.

It’d be fun to actually calculate this efficiency. My local power is mostly nuclear so I wonder how that works out.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#172
post #155

One clever ingredient in OpenAI's secret sauce is billions of dollars of losses. About $5 billion dollars lost in 2024. https://www.cnbc.com/2024/09/27/openai-sees-5-billion-loss-t...

they would be break-even if all they did was serve existing models and got rid of everything related to R&D

Have they considered replacing their engineers with AI?

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#173

Earlier quoted context omitted.

You can knock off a zero or two just by time shifting the 700 million distinct users across a day/week and account for the mere minutes of compute time they will actually use in each interaction. So they might no see peaks higher than 10 million active inference session at the same time. Conversely, you can't do the same thing as a self hosted user, you can't really bank your idle compute for a week and consume it al…

During times of high utilization, how do they handle more requests than they have hardware? Is the software granular enough that they can round robin the hardware per token generated? UserA token, then UserB, then UserC, back to UserA? Or is it more likely that everyone goes into a big FIFO processing the entire request before switching to the next user? I assume the former has massive overhead, but maybe it is worth…

This is great product design at its finest.

First of all, they never “handle more requests than they have hardware.” That’s impossible (at least as I’m reading it).

The vast majority of usage is via their web app (and free accounts, at that). The web app defaults to “auto” selecting a model. The algorithm for that selection is hidden information.

As load peaks, they can divert requests to different levels of hardware and less resource hungry models.

Only a very small minority of requests actually specify the model to use.

There are a hundred similar product design hacks they can use to mitigate load. But this seems like the easiest one to implement.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#174

Earlier quoted context omitted.

During times of high utilization, how do they handle more requests than they have hardware? Is the software granular enough that they can round robin the hardware per token generated? UserA token, then UserB, then UserC, back to UserA? Or is it more likely that everyone goes into a big FIFO processing the entire request before switching to the next user? I assume the former has massive overhead, but maybe it is worth…

Inference is essentially a very complex matrix algorithm run repeatedly on itself, each time the input matrix (context window) is shifted and the new generated tokens appended to the end. So, it's easy to multiplex all active sessions over limited hardware, a typical server can hold hundreds of thousands of active contexts in the main system ram, each less than 500KB and ferry them to the GPU nearly instantaneously a…

I was under the impression that context takes up a lot more VRAM than this.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#175
post #109

Earlier quoted context omitted.

Someone's take on AI was that we're collectively investing billions in data centers that will be utterly worthless in 10 years. Unlike the investments in railways or telephone cables or roads or any other sort of architecture, this investment has a very short lifespan. Their point was that whatever your take on AI, the present investment in data centres is a ridiculous waste and will always end up as a huge net loss…

The servers may well be worthless (or at least worth a lot less), but that's pretty much true for a long time. Not many people want to run on 10 year old servers (although I pay $30/month for a dedicated server that's dual Xeon L5640 or something like that, which is about 15 years old). The servers will be replaced, the networking equipment will be replaced. The building will still be useful, the fiber that was pulle…

I have a server in my office that's at from 2009 still far more economical to run than buying any sort of cloud compute. By at least an order of magnitude.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#176
post #159

Earlier quoted context omitted.

Opt if you ignore that both gas furnaces and heat pumps are more efficient than resistive loads.

I'm in the market for an oven right now and 230V/16A is the voltage/current the one I'll probably be getting operates under. At 90°C you can do sous vide, so basically use that waste heat entirely. For such temperatures you'd need a CO2 heat pump, which is still expensive. I don't know about gas, as I don't even have a line to my place.

How can you bear to eat sous vide though? I've tried it for months and years, and I still find it troublesome. So mushy, nothing enjoy.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#177

One trick is that multiple prompts are combined into a batch that is subject to the token prediction simultaneously. I've had Google Gemini start talking to be in the context of someone else's chat, unrelated to mine. ME: Does POSIX have any API for creating a filesystem object (such as a file) with a predetermined inode number supplied by the caller? AI: \documentclass{article} \usepackage{amsmath} \usepackage{amsfo…

> There was no such instruction. I've never chatted with any AI about LaTeX. it leaked from the tokens of someone else's chat.

Nope. That's not how it works. Attention doesn't work across multiple independent prompts queued in the same batch. It's not physically possible for the tokens of another chat to leak.

What most likely happened is that the model glitched out to the instructions in its (hidden) system prompt, which most likely does include instructions about using LaTeX for mathematical and scientific notation.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#178

Earlier quoted context omitted.

What do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.

It's not waste heat if you only run it in the winter.

Seasonality in git commit frequency

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#179

An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper" I don't think people realize the size of these compute u…

As a non-American the 240V thing made me laugh.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#180

Earlier quoted context omitted.

A lot of really smart people working on problems that don't even really need to be solved is an interesting aspect of market allocation.

Can you explain what you mean about 'not needing to be solved'? There are versions of that kind of critique that would seem, at least on the surface, to better apply to finance or flash trading. I ask because scaling an system that a substantially chunk of the population finds incredibly useful, including for the more efficient production of public goods (scientific research, for example) does seem like a problem tha…

I think the problem I see with this type of response is that it doesn't take into context the waste of resources involved. If the 700M users per week is legitimate then my question to you is: how many of those invocations are worth the cost of resources that are spent, in the name of things that are truly productive?

And if AI was truly the holy grail that it's being sold as then there wouldn't be 700M users per week wasting all of these resources as heavily as we are because generative AI would have already solved for something better. It really does seem like these platforms are, and won't be, anywhere as useful as they're continuously claimed to be.

Just like Tesla FSD, we keep hearing about a "breakaway" model and the broken record of AGI. Instead of getting anything exceptionally better we seem to be getting models tuned for benchmarks and only marginal improvements.

I really try to limit what I'm using an LLM for these days. And not simply because of the resource pigs they are, but because it's also often a time sink. I spent an hour today testing out GPT-5 and asking it about a specific problem I was solving for using only 2 well documented technologies. After that hour it had hallucinated about a half dozen assumptions that were completely incorrect. One so obvious that I couldn't understand how it had gotten it so wrong. This particular technology, by default, consumes raw SSE. But GPT-5, even after telling it that it was wrong, continued to give me examples that were in a lot of ways worse and kept resorting to telling me to validate my server responses were JSON formatted in a particularly odd way.

Instead of continuing to waste my time correcting the model I just went back to reading the docs and GitHub issues to figure out the problem I was solving for. And that led me down a dark chain of thought: so what happens when the "teaching" mode rethinks history, or math fundamentals?

I'm sure a lot of people think ChatGPT is incredibly useful. And a lot of people are bought into not wanting to miss the boat, especially those who don't have any clue to how it works and what it takes to execute any given prompt. I actually think LLMs have a trajectory that will be similar to social media. The curve is different and I, hopefully, don't think we've seen the most useful aspects of it come to fruition as of yet. But I do think that if OpenAI is serving 700M users per week then, once again, we are the product. Because if AI could actually displace workers en masse today you wouldn't have access to it for $20/month. And they wouldn't offer it to you at 50% off for the next 3 months when you go to hit the cancel button. In fact, if it could do most of the things executives are claiming then you wouldn't have access to it at all. But, again, the users are the product - in very much the same way social media played into.

Finally, I'd surmise that of those 700M weekly users less than 10% of those sessions are being used for anything productive that you've mentioned and I'd place a high wager that the 10% is wildly conservative. I could be wrong, but again - we'd know about that if it were the actual truth.

Post reply on HN