Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

371–379 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#371

Post by DoctorOetker »

dbbk wrote: post #361
DoctorOetker wrote: post #307

Earlier quoted context omitted.

Due to batching, inference is profitable, very profitable. Yet undoubtedly they are making what is declared a loss. But is it really a loss? If you buy an asset, is that automatically a loss? or is it an investment? By "running at a loss" one can build a huge dataset, to stay in the running.

How batched can it really be though if every request is personalised to the user with Memory?

Imagine pipelineing lots of infra-scale GPU's, naive inference would need all previous tokens to be shifted "left" or from the append-head to the end-of-memory "tail", which would require a huge amount of data flow for the whole KV cache etc. Instead of calling GPU 1 the end-of-memory and GPU N the append-head, you keep the data static and let the role rotate like a circular buffer. So now for each new token inference round, the previous rounds end-of-memory GPU becomes the new append-head GPU. The highest bandwidth is keeping data static.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#373

Post by frabcus »

nodja wrote: post #297
frabcus wrote: post #255

Earlier quoted context omitted.

This seems a bit complicated to me. They don't serve very many models. My assumption is they just dedicate GPUs to specific models, so the model is always in VRAM. No loading per request - it takes a while to load a model in anyway. The limiting factor compared to local is dedicated VRAM - if you dedicate 80GB of VRAM locally 24 hours/day so response times are fast, you're wasting most of the time when you're not que…

Loading here refers to loading from VRAM to the GPUs core cache, loading from VRAM is extremely slow in terms of GPU time that GPU cores end up idle most of the time just waiting for more data to come in.

Thanks, got it! Think I need a deeper article on this - as comment below says you'd then need to load the request specific state in instead.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#374

Post by nulltype »

semi-extrinsic wrote: post #38
airhangerf15 wrote: post #11

An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper" I don't think people realize the size of these compute u…

What do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.

> What do you mean 10 years?

Didn’t the DGX-1 come out 9 years ago?

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#375

Post by throwaway2037 »

brookst wrote: post #367
kossTKR wrote: post #350

Earlier quoted context omitted.

Really? So the system recognises someone asked the same question and serves the same answer? And who on earth shares the exact same context? I mean i get the idea but sounds so incredibly rare it would mean absolutely nothing optimisation wise.

Yes. It is not incredibly rare, it's incredibly common. A huge percentage of queries to retail LLMs are things like "hello" and "what can you do", with static system prompts that make the total context identical. It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service.

    > It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service.
So if 3% is 500M, then annual spend is ~16.6B. That is medium sized these days?

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#376

Post by throwaway2037 »

serf wrote: post #138
scarface_74 wrote: post #111

Earlier quoted context omitted.

It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP. But then again even there, their reputation for abandoning products, lack of customer service, condescension when it came to large enterprises’ “legacy tech” lets Microsoft who is king of hand holding big enterprise and even AWS run rough shod over them. When I was at AWS ProServe, we didn’t even bother coming up with talki…

>It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP. there are few groups as talented at losing a head start as google.

DEC, Sun, Intel, Blackberry, Motorola, Nokia: More talented in that area!

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#377

Post by throwaway2037 »

0_____0 wrote: post #293
benreesman wrote: post #262

Earlier quoted context omitted.

Google will win the LLM game if the LLM game is about compute , which is the common wisdom and maybe true, but not foreordained by God. There's an argument that if compute was the dominant term that Google would never have been anything but leading by a lot. Personally right now I see one clear leader and one group going 0-99 like a five sigma cosmic ray: Anthropic and the PRC. But this is because I believe/know that…

I believe Google might win the LLM game simply because they have the infrastructure to make it profitable - via ads . All the LLM vendors are going to have to cope with the fact that they're lighting money on fire, and Google have the paying customers (advertisers) and with the user-specific context they get from their LLM products, one of the juciest and most targetable ad audiences of all time.

This is one of the best insights after reading 100+ posts here. You are talking about existing demand and existing relationships from their advert business. These customers are happy with Google results. The Google marketing team will be carefully defining new advert products that employ LLMs.

I would only offer one disagreement with your post: There will not be a single winner in LLMs. The landscape is so large that we will have multiple winners in different areas. Example: Google might fail in B2C LLM (chatbot that answers your questions), but will certainly be (wildly?) successful in B2B for adverts.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#378

Post by 0_____0 »

throwaway2037 wrote: post #377
0_____0 wrote: post #293

Earlier quoted context omitted.

I believe Google might win the LLM game simply because they have the infrastructure to make it profitable - via ads . All the LLM vendors are going to have to cope with the fact that they're lighting money on fire, and Google have the paying customers (advertisers) and with the user-specific context they get from their LLM products, one of the juciest and most targetable ad audiences of all time.

This is one of the best insights after reading 100+ posts here. You are talking about existing demand and existing relationships from their advert business. These customers are happy with Google results. The Google marketing team will be carefully defining new advert products that employ LLMs. I would only offer one disagreement with your post: There will not be a single winner in LLMs. The landscape is so large that…

Hm, yes I agree that winner-take-all is not necessary. A viable monetization strategy is. Meta and Google have this through their ad business, and I think it's possible that some other smaller players might bring in revenue by targeting particular niches like industrial and defense customers who need an LLM product but have data security needs and can afford the product directly.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#379

Post by parentheses »

0cf8612b2e1e wrote: post #56
cornholio wrote: post #43

Earlier quoted context omitted.

You can knock off a zero or two just by time shifting the 700 million distinct users across a day/week and account for the mere minutes of compute time they will actually use in each interaction. So they might no see peaks higher than 10 million active inference session at the same time. Conversely, you can't do the same thing as a self hosted user, you can't really bank your idle compute for a week and consume it al…

During times of high utilization, how do they handle more requests than they have hardware? Is the software granular enough that they can round robin the hardware per token generated? UserA token, then UserB, then UserC, back to UserA? Or is it more likely that everyone goes into a big FIFO processing the entire request before switching to the next user? I assume the former has massive overhead, but maybe it is worth…

They probably do lots of tricks like using quantized or distilled models during times of high load. They also have a sizeable number of free users, who will be the first to get rate limited.
Post reply on HN