Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

251–260 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#251

Post by tgma »

mlyle wrote: post #228
tgma wrote: post #195

Earlier quoted context omitted.

Sure. That has nothing to do with the efficiency of your system though. As far as you are concerned this is about your electricity consumption for the home server vs gas consumption. In that sense resistive heat inside the home is 100% efficient compared to gas furnace; the fuel cost might be lower on the latter.

Sure, it's "equally efficient" if you ignore the inefficient thing that is done outside where you draw the system box, directly in proportion to how much you do it. Heating my house with a giant diesel-powered radiant heater from across the street is infinitely efficient, too, since I use no power in my house.

If you don’t close the box of the system at some point to isolate the input, efficiency would be meaningless. I think in the context of the original post, suggesting running a server in winter would be a zero-waste endeavor if you need the heat anyway, it is perfectly clear that the input is electricity to your home at a certain $/kWh and gas at a certain $/BTU. Under that premise, it is fair to say that would not be true if you have a heat pump deployed but would be true compared to gas furnace in terms of efficiency (energy consumed for unit of heat), although not necessarily true economically.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#252

Post by cornholio »

apitman wrote: post #174
cornholio wrote: post #73

Earlier quoted context omitted.

Inference is essentially a very complex matrix algorithm run repeatedly on itself, each time the input matrix (context window) is shifted and the new generated tokens appended to the end. So, it's easy to multiplex all active sessions over limited hardware, a typical server can hold hundreds of thousands of active contexts in the main system ram, each less than 500KB and ferry them to the GPU nearly instantaneously a…

I was under the impression that context takes up a lot more VRAM than this.

The context after application of the algorithm is just text, something like 256k input tokens, each token representing a group of roughly 2-5 characters, encoded into 18-20 bits.

The active context during inference, inside the GPUs, explodes each token into a 12288 dimensions vector, so 4 orders of magnitude more VRAM, and is combined with the model weights, Gbytes in size, across multiple parallel attention heads. The final result are just more textual tokens, which you can easily ferry around main system RAM and send to the remote user.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#255

Post by frabcus »

superasn wrote: post #234
nodja wrote: post #222

Earlier quoted context omitted.

This is the real answer, I don't know what people above are even discussing when batching is the biggest reduction in costs. If it costs say $50k to serve one request, with batching is also costs $50k to serve 100 at the same time with minimal performance loss, I don't know what the real number of users is before you need to buy new hardware, but I know it's in the hundreds so going from $50000 to $500 in effective c…

Thanks for the helpful reply! As I wasn't able to fully understand it still, I pasted your reply in chatgpt and asked it some follow up questions and here is what i understand from my interaction: - Big models like GPT-4 are split across many GPUs (sharding). - Each GPU holds some layers in VRAM. - To process a request, weights for a layer must be loaded from VRAM into the GPU's tiny on-chip cache before doing the ma…

This seems a bit complicated to me. They don't serve very many models. My assumption is they just dedicate GPUs to specific models, so the model is always in VRAM. No loading per request - it takes a while to load a model in anyway.

The limiting factor compared to local is dedicated VRAM - if you dedicate 80GB of VRAM locally 24 hours/day so response times are fast, you're wasting most of the time when you're not querying.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#256

Post by kj4ips »

iJohnDoe wrote: post #224
kj4ips wrote: post #77

Earlier quoted context omitted.

Fair warning: the BMCs on those suck so bad, and the firmware bundles are painful, since you need a working nvidia-specific container runtime to apply them, which you might not be able to get up and running because of a firmware bug causing almost all the ram to be presented as nonvolatile.

Are there better paths you would suggest? Any hardware people have reported better luck with?

Honestly, unless you //really// need nvlink/ib (meaning that copies and pcie trips are your bottleneck), you may do better with whatever commodity system with sufficient lanes, slots, and CFM is available at a good price.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#257

Post by patrick451 »

zozbot234 wrote: post #248
KaiserPro wrote: post #95

Earlier quoted context omitted.

Same explanation but with less mysticism: Inference is (mostly) stateless. So unlike training where you need to have memory coherence over something like 100k machines and somehow avoid the certainty of machine failure, you just need to route mostly small amounts of data to a bunch of big machines. I don't know what the specs of their inference machines are, but where I worked the machines research used were all 8gpu…

> Inference is (mostly) stateless. ... you just need to route mostly small amounts of data to a bunch of big machines. I think this might just be the key insight. The key advantage of doing batched inference at a huge scale is that once you maximize parallelism and sharding, your model parameters and the memory bandwidth associated with them are essentially free (since at any given moment they're being shared among a…

> The key advantage of doing batched inference at a huge scale is that once you maximize parallelism and sharding, your model parameters and the memory bandwidth associated with them are essentially free (since at any given moment they're being shared among a huge amount of requests!)

Kind of unrelated, but this comment made me wonder when we will start seeing side channel attacks that force queries to leak into each other.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#258

Post by nocturnes »

addaon wrote: post #239
computomatic wrote: post #173

Earlier quoted context omitted.

This is great product design at its finest. First of all, they never “handle more requests than they have hardware.” That’s impossible (at least as I’m reading it). The vast majority of usage is via their web app (and free accounts, at that). The web app defaults to “auto” selecting a model. The algorithm for that selection is hidden information. As load peaks, they can divert requests to different levels of hardware…

> But this seems like the easiest one to implement. Even easier: Just fail. In my experience the ChatGPT web page fails to display (request? generate?) a response between 5% and 10% of the time, depending on time of day. Too busy? Just ignore your customers. They’ll probably come back and try again, and if not, well, you’re billing them monthly regardless.

Is this a common experience for others? In several years of reasonable ChatGPT use I have only experienced that kind of failure a couple of times.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#259

Post by Horos »

ActorNightly wrote: post #97
fakedang wrote: post #59

Earlier quoted context omitted.

Yeah honestly. They could just try selling solutions and SLAs combining their TPU hardware with on-prem SOTA models and practically dominate enterprise. From what I understand, that's GCP's gameplay too for most regulated enterprise clients.

Googles bread and butter is advertising, so they have a huge interest in keeping things in house. Data is more valuable to them than money from hardware sales. Even then, I think that their primary use case is going to be consumer grade good AI on phones. I dunno why Gemma QAT model fly so low on the radar, but you can basically get full scale Llamma 3 like performance from a single 3090 now, at home.

Gemma Term of uses ?

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#260

Post by hnpolicestate »

canyon289 wrote: post #6

I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast. https://jax-ml.git…

This caught my attention "But today even “small” models run so close to hardware limits".

Sounds analogous to the 60's and 70's i.e "even small programs run so close to hardware limits". If optimization and efficiency is dead in software engineering, it's certainly alive and well in LLM development.

Post reply on HN