Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

311–320 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#311

Post by thechao »

rlupi wrote: post #292
scarface_74 wrote: post #111

Earlier quoted context omitted.

It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP. But then again even there, their reputation for abandoning products, lack of customer service, condescension when it came to large enterprises’ “legacy tech” lets Microsoft who is king of hand holding big enterprise and even AWS run rough shod over them. When I was at AWS ProServe, we didn’t even bother coming up with talki…

> It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP. > But then again even there, their reputation for abandoning products What are the chances of abandoning TPU-related projects where the company literally invested billions in infrastructure? Zero.

I don't know what scale of "billions" you're talking about; but, Intel blew 1–2 billion on Larrabee. Even worse: Intel blew 5+ billion on mobile pre-iPhone. I remember when that team was shown the door — that's when we had to evaluate the early RGX GPUs as a backstop to try to win Apple's business; the RGX's were turds.

Penny-wise pound-foolish.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#312

Post by jsnell »

ritz_labringue wrote: post #86

The short answer is "batch size". These days, LLMs are what we call "Mixture of Experts", meaning they only activate a small subset of their weights at a time. This makes them a lot more efficient to run at high batch size. If you try to run GPT4 at home, you'll still need enough VRAM to load the entire model, which means you'll need several H100s (each one costs like $40k). But you will be under-utilizing those card…

> These days, LLMs are what we call "Mixture of Experts", meaning they only activate a small subset of their weights at a time. This makes them a lot more efficient to run at high batch size.

I don't really understand why you're trying to connect MoE and batching here. Your stated mechanism is not only incorrect but actually the wrong way around.

The efficiency of batching comes from optimally balancing the compute and memory bandwidth, by loading a tile of parameters from the VRAM to cache, applying those weights to all the batched requests, and only then loading in the next tile.

So batching only helps when multiple queries need to access the same weights for the same token. For dense models, that's just what always happens. But for MoE, it's not the case, exactly due to the reason that not all weights are always activated. And then suddenly your batching becomes a complex scheduling problem, since not all the experts at a given layer will have the same load. Surely a solvable problem, but MoE is not the enabler for batching but making it significantly harder.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#313

Post by xtiansimon »

semi-extrinsic wrote: post #38
airhangerf15 wrote: post #11

An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper" I don't think people realize the size of these compute u…

What do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.

> “They are heavy, noisy like you would not believe, … produces … waste heat.”

Haha. I bought a 20 yro IBM server off eBay for a song. It was fun for a minute. Soon became a doorstop and I sold it as pickup-only on eBay for $20. Beast. Never again have one in my home.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#314

Post by claytongulick »

blibble wrote: post #137
canyon289 wrote: post #6

I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast. https://jax-ml.git…

> So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. "we do 1970s mainframe style timesharing" there, that was easy

I'm pretty sure that's not right.

They're definitely running cluster knoppix.

:-)

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#315

Post by jon-wood »

abletonlive wrote: post #161
YossarianFrPrez wrote: post #154

Earlier quoted context omitted.

Can you explain what you mean about 'not needing to be solved'? There are versions of that kind of critique that would seem, at least on the surface, to better apply to finance or flash trading. I ask because scaling an system that a substantially chunk of the population finds incredibly useful, including for the more efficient production of public goods (scientific research, for example) does seem like a problem tha…

They won’t be honest and explain it to you but I will. Takes like the one you’re responding to are from loathsome pessimistic anti-llm people that are so far detached from reality they can just confidently assert things that have no bearing on truth or evidence. It’s a coping mechanism and it’s basically a prolific mental illness at this point

> so far detached from reality they can just confidently assert things that have no bearing on truth or evidence

So not unlike an LLM then?

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#316

Post by windexh8er »

mlyle wrote: post #226
windexh8er wrote: post #180

Earlier quoted context omitted.

I think the problem I see with this type of response is that it doesn't take into context the waste of resources involved. If the 700M users per week is legitimate then my question to you is: how many of those invocations are worth the cost of resources that are spent, in the name of things that are truly productive? And if AI was truly the holy grail that it's being sold as then there wouldn't be 700M users per week…

> If the 700M users per week is legitimate then my question to you is: how many of those invocations are worth the cost of resources that are spent, in the name of things that are truly productive? Is everything you spend resources on truly productive? Who determines whether something is worth it? Is price/willingness of both parties to transact not an important factor? I don't think ChatGPT can do most things I do.…

I don't believe everything in my world is as efficient as it could be. But I genuinely think about the costs involved [0]. When doing automations that are perfectly handled by deterministic systems why would I put the outcomes of those in the hands of a non-deterministic one? And at that cost differential?

We know a few things: LLMs are not efficient, LLMs are consuming more water than traditional compute, we know the providers know but they haven't shared any tangible metrics, and the build process involves, also, an exceptional amount of time, wattage and water.

For me it's: if you have access to a supercomputer do you use it to tell you a joke or work on a life saving medicine?

We didn't have these tools 5 years ago. 5 years ago you dealt with said "drudgery". On the other hand you then say it can't do "most things I do". It seems as though the lines of fatalism and paradox are in full force for a lot of the arguments around AI.

I think the real kicker for me this week (and it changes week-over-week, which is at least entertaining) is when Paul Graham told his Twitter feed [1] a "hotshot" programmer is writing 10k LOC that are not "bug-filled crap" in 12 hours. That's 14 LOC per minute. Compared to industry norms of 50-150 LOC per 8 hour day. Apparently,this "hot-shot" is not "naive", though, implying that it's most definitely legit.

[0] https://www.sciencenews.org/article/ai-energy-carbon-emissio... [1] https://x.com/paulg/status/1953289830982664236

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#318

Post by bawana »

hirvi74 wrote: post #192
windexh8er wrote: post #180

Earlier quoted context omitted.

I think the problem I see with this type of response is that it doesn't take into context the waste of resources involved. If the 700M users per week is legitimate then my question to you is: how many of those invocations are worth the cost of resources that are spent, in the name of things that are truly productive? And if AI was truly the holy grail that it's being sold as then there wouldn't be 700M users per week…

> so what happens when the "teaching" mode rethinks history, or math fundamentals? The person attempting to learn either (hopefully) figures out the AI model was wrong, or sadly learns the wrong material. The level of impact is probably quite relative to how useful the knowledge is one's life. The good or bad news, depending on how you look at it, is that humans are already great at rewriting history and believing wr…

Reminds me of our president

https://www.bbc.com/news/world-us-canada-52407177.amp

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#319

Post by bawana »

nodja wrote: post #296
superasn wrote: post #234

Earlier quoted context omitted.

Thanks for the helpful reply! As I wasn't able to fully understand it still, I pasted your reply in chatgpt and asked it some follow up questions and here is what i understand from my interaction: - Big models like GPT-4 are split across many GPUs (sharding). - Each GPU holds some layers in VRAM. - To process a request, weights for a layer must be loaded from VRAM into the GPU's tiny on-chip cache before doing the ma…

Yeah chatgpt pretty much nailed it.

But you still have to load the data for each request. And in an LLM doesnt this mean the WHOLE kv cache because the kv cache changes after every computation? So why isnt THIS the bottleneck? Gemini is talking about a context window of a million tokens- how big would the kv cache fir this get?

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#320

Post by bawana »

joshhart wrote: post #250

A single node with GPUs has a lot of FLOPs and very high memory bandwidth. When only processing a few requests at a time, the GPUs are mostly waiting on the model weights to stream from the GPU ram to the processing units. When batching requests together, they can stream a group of weights and score many requests in parallel with that group of weights. That allows them to have great efficiency. Some of the other main…

How is speculative decoding helpful if you still have to run the full model against which you check the results?
Post reply on HN