Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

151–160 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#151

Post by CamperBob2 »

oceanplexian wrote: post #147
CamperBob2 wrote: post #72

Earlier quoted context omitted.

Are you talking about the guy in Temecula running two different auctions with some of the same photos (356878140643 and 357146508609, both showing a missing heat sink?) Interesting, but seems sketchy. How useful is this Tesla-era hardware on current workloads? If you tried to run the full DeepSeek R1 model on it at (say) 4-bit quantization, any idea what kind of TTFT and TPS figures might be expected?

I can’t speak to the Tesla stuff but I run an Epyc 7713 with a single 3090 and creatively splitting the model between GPU/8 channels of DDR4 I can do about 9 tokens per second on a q4 quant.

Impressive. Is that a distillation, or the real thing?

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#152

Post by vermilingua »

catigula wrote: post #148
canyon289 wrote: post #6

I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast. https://jax-ml.git…

A lot of really smart people working on problems that don't even really need to be solved is an interesting aspect of market allocation.

Well, we all thought advertising was the worst thing to come out of the tech industry, someone had to prove us wrong!

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#153

Post by edoceo »

davedx wrote: post #110
ActorNightly wrote: post #28

Earlier quoted context omitted.

Yes. Google is probably gonna win the LLM game tbh. They had a massive head start with TPUs which are very energy efficient compared to Nvidia Cards.

But they’re ASICs so any big architecture changes will be painful for them right?

I'd think no. They have the hardware and software experience, likely have next and next-next plans in place already. The big hurdle is money, which G has a bunch of.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#154

Post by YossarianFrPrez »

catigula wrote: post #148
canyon289 wrote: post #6

I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast. https://jax-ml.git…

A lot of really smart people working on problems that don't even really need to be solved is an interesting aspect of market allocation.

Can you explain what you mean about 'not needing to be solved'? There are versions of that kind of critique that would seem, at least on the surface, to better apply to finance or flash trading.

I ask because scaling an system that a substantially chunk of the population finds incredibly useful, including for the more efficient production of public goods (scientific research, for example) does seem like a problem that a) needs to be solved from a business point of view, and b) should be solved from a civic-minded point of view.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#155

Post by 93po »

abathologist wrote: post #63

One clever ingredient in OpenAI's secret sauce is billions of dollars of losses. About $5 billion dollars lost in 2024. https://www.cnbc.com/2024/09/27/openai-sees-5-billion-loss-t...

they would be break-even if all they did was serve existing models and got rid of everything related to R&D

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#156

Post by wisty »

They can have a very even load if they use their nodes for training when the customer use is low, so that massively helps. If they have 3x as much hardware as they need to serve peak demand (even with throttling) this will cost a lot, unless they have a another use for lots of GPU.

Just illustrative guesses, not real numbers, I underestimate overheads here but anyway ...

Let's assume a $20k expert node can produce 500 tokens per second (15,000 per year). $5k a year for the machine per year. $5k overheads. 5 experts per token (so $50k to produce 15,000 megatokens with a 100% throughput). Say they charge up to $10 per million tokens ... yeah it's tight but I can see how it's doable.

Say they cost $100 per user per year. If it's $10 per million tokens (depends on the model) then they are budgeting 10 million tokens per user. That's like 100 books per year. The answer is that users probably don't use as much as the api would cost.

The real question is, how does it cost $10 per megatoken?

500 tokens per second per node is like 15,000 megatokens per year. So a 500 token node can bring in $150,000 per node.

Call it 5 live experts and a router. That's maybe $20k per expert per year. If it's a kilowatt power supply per expert, and $0.1 per kW power that's $1000 for power. The hardware is good for 4 years so $5k for that. Toss in overheads, and it's maybe $10k costs.

So at full capacity they can make $5 off $10 revenue. With uneven loads they make nothing, unless they have some optimisation and very good load balancing (if they can double the tokens per second then they make a decent profit).

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#157

Post by catigula »

YossarianFrPrez wrote: post #154
catigula wrote: post #148

Earlier quoted context omitted.

A lot of really smart people working on problems that don't even really need to be solved is an interesting aspect of market allocation.

Can you explain what you mean about 'not needing to be solved'? There are versions of that kind of critique that would seem, at least on the surface, to better apply to finance or flash trading. I ask because scaling an system that a substantially chunk of the population finds incredibly useful, including for the more efficient production of public goods (scientific research, for example) does seem like a problem tha…

[flagged]

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#159

Post by Tade0 »

hdgvhicv wrote: post #116
ksherlock wrote: post #90

Earlier quoted context omitted.

It's not waste heat if you only run it in the winter.

Opt if you ignore that both gas furnaces and heat pumps are more efficient than resistive loads.

I'm in the market for an oven right now and 230V/16A is the voltage/current the one I'll probably be getting operates under.

At 90°C you can do sous vide, so basically use that waste heat entirely.

For such temperatures you'd need a CO2 heat pump, which is still expensive. I don't know about gas, as I don't even have a line to my place.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#160

Post by arjvik »

ritz_labringue wrote: post #86

The short answer is "batch size". These days, LLMs are what we call "Mixture of Experts", meaning they only activate a small subset of their weights at a time. This makes them a lot more efficient to run at high batch size. If you try to run GPT4 at home, you'll still need enough VRAM to load the entire model, which means you'll need several H100s (each one costs like $40k). But you will be under-utilizing those card…

Essentially, inference is well-amortized across the many users.
Post reply on HN