Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

301–310 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#301

Post by jeltz »

gregorygoc wrote: post #299
jeltz wrote: post #295

Earlier quoted context omitted.

I have met those supersmart specialists but in my experience there are also a lot of smart people who are more generalists. The real answer is likely internal company politics and priorities. Google certainly has people with the technical skills to solve it but do they care and if they care can they allocate those skilled people to the task?

My observation is that in general smart generalists are smarter than smart specialists. I work at Google, and it’s just that these generalists folks are extremely fast learners. They can cover breadth and depth of an arbitrary topic in a matter of 15 minutes, just enough to solve a problem at hand. It’s quite intimidating how fast they can break down difficult concepts into first principles. I’ve witnessed this first…

That is my experience too. It sometimes seem the supersmart generalists are people whose strongest skill is learning.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#302

Post by zozbot234 »

saagarjha wrote: post #282
zozbot234 wrote: post #248

Earlier quoted context omitted.

> Inference is (mostly) stateless. ... you just need to route mostly small amounts of data to a bunch of big machines. I think this might just be the key insight. The key advantage of doing batched inference at a huge scale is that once you maximize parallelism and sharding, your model parameters and the memory bandwidth associated with them are essentially free (since at any given moment they're being shared among a…

mmap is not free. It just moves bandwidth around.

Using mmap for model parameters allows you to run vastly larger models for any given amount of system RAM. It's especially worthwhile when you're running MoE models and parameters for unused "experts" can just be evicted from RAM, leaving room for more relevant data. But of course this applies more generally to, e.g. single model layers, etc.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#303

Post by Lionga »

knowitnone2 wrote: post #237
93po wrote: post #155

Earlier quoted context omitted.

they would be break-even if all they did was serve existing models and got rid of everything related to R&D

they are not the only player so getting rid of R&D would be suicide

It is now 3 years in where I was told AI will replace engineers in 6 month. How come all the AI companies have not replaced engineers?

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#305

Post by justincormack »

CamperBob2 wrote: post #72
semi-extrinsic wrote: post #38

Earlier quoted context omitted.

What do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.

Are you talking about the guy in Temecula running two different auctions with some of the same photos (356878140643 and 357146508609, both showing a missing heat sink?) Interesting, but seems sketchy. How useful is this Tesla-era hardware on current workloads? If you tried to run the full DeepSeek R1 model on it at (say) 4-bit quantization, any idea what kind of TTFT and TPS figures might be expected?

Tesla doesnt support 4 bit float.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#306

Post by qcnguy »

adastra22 wrote: post #210
baxtr wrote: post #91

Earlier quoted context omitted.

The only one who can stop Google is Google. They’ll definitely have the best model, but there is a chance they will f*up the product / integration into their products.

There is plenty of time left to fumble the ball.

And they already did many times.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#307

Post by DoctorOetker »

abathologist wrote: post #63

One clever ingredient in OpenAI's secret sauce is billions of dollars of losses. About $5 billion dollars lost in 2024. https://www.cnbc.com/2024/09/27/openai-sees-5-billion-loss-t...

Due to batching, inference is profitable, very profitable.

Yet undoubtedly they are making what is declared a loss.

But is it really a loss?

If you buy an asset, is that automatically a loss? or is it an investment?

By "running at a loss" one can build a huge dataset, to stay in the running.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#309

Post by qcnguy »

tough wrote: post #20
roadside_picnic wrote: post #12

I'm sure there are countless tricks, but one that can implemented at home, and I know plays a major part in Cerebras' performance is: speculative decoding. Speculative decoding uses a smaller draft model to generate tokens with much less compute and memory required. Then the main model will accept those tokens based on the probability it would have generated them. In practice this case easily result in a 3x speedup i…

gpt-oss-120b can be used with gpt-oss-20b as speculative drafting on LM Studio I'm not sure it improved the speed much

It depends a lot on the type of conversation. A lot of ChatGPT load appears to be therapy talk that even small models can correctly predict.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#310

Post by abdullin »

KaiserPro wrote: post #95
canyon289 wrote: post #6

I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast. https://jax-ml.git…

Same explanation but with less mysticism: Inference is (mostly) stateless. So unlike training where you need to have memory coherence over something like 100k machines and somehow avoid the certainty of machine failure, you just need to route mostly small amounts of data to a bunch of big machines. I don't know what the specs of their inference machines are, but where I worked the machines research used were all 8gpu…

> Inference is (mostly) stateless

Quite the opposite. Context caching requires state (K/V cache) close to the VRAM. Streaming requires state. Constrained decoding (known as Structured Outputs) also requires state.

Post reply on HN