Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#121

Post by hdgvhicv »

simonw wrote: post #46
jl6 wrote: post #41

If the explanation really is, as many comments here suggest, that prompts can be run in parallel in batches at low marginal additional cost, then that feels like bad news for the democratization and/or local running of LLMs. If it’s only cost-effective to run a model for ~thousands of people at the same time, it’s never going to be cost-effective to run on your own.

Sure, but that's how most of human society works already. It's more cost effective to farm eggs from a hundred thousand chickens than it is for individuals to have chickens in their yard. You CAN run a GPT-class model on your own machine right now, for several thousand dollars of machine... but you can get massively better results if you spend those thousands of dollars on API credits over the next five years or so.…

50 years ago general computers were also time shared. Then the pendulum swing to desktop, then back to central.

I for one look forward to another 10 years of progress - or less - putting current models running on a laptop. I don’t trust any big company with my data

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#123

Post by anon291 »

At the end of the day, the answer is... specialized hardware. No matter what you do on your local system, you don't have the interconnects necessary. Yes, they have special software, but the software would not work locally. NVIDIA sells entire solutions and specialized interconnects for this purpose. They are well out of the reach of the standard consumer.

But software wise, they shard, load balance, and batch. ChatGPT gets 1000s (or something like that) of requests every second. Those are batched and submitted to one GPU. Generating text for 1000 answers is often the same speed as generating for just 1 due to how memory works on these systems.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#124

Post by tgma »

hdgvhicv wrote: post #116
ksherlock wrote: post #90

Earlier quoted context omitted.

It's not waste heat if you only run it in the winter.

Opt if you ignore that both gas furnaces and heat pumps are more efficient than resistive loads.

Heat pump sure, but how is gas furnace more efficient than resistive load inside the house? Do you mean more economical rather than more efficient (due to gas being much cheaper/unit of energy)?

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#125

Post by Skunkleton »

eulgro wrote: post #106
semi-extrinsic wrote: post #38

Earlier quoted context omitted.

What do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.

> 13 000 BTU/hr In sane units: 3.8 kW

> In sane units: 3.8 kW

5.1 Horsepower

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#126

Post by [deleted] »

davedx wrote: post #110
ActorNightly wrote: post #28

Earlier quoted context omitted.

Yes. Google is probably gonna win the LLM game tbh. They had a massive head start with TPUs which are very energy efficient compared to Nvidia Cards.

But they’re ASICs so any big architecture changes will be painful for them right?

[deleted]

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#127

Post by [deleted] »

scarface_74 wrote: post #111
baxtr wrote: post #91

Earlier quoted context omitted.

The only one who can stop Google is Google. They’ll definitely have the best model, but there is a chance they will f*up the product / integration into their products.

It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP. But then again even there, their reputation for abandoning products, lack of customer service, condescension when it came to large enterprises’ “legacy tech” lets Microsoft who is king of hand holding big enterprise and even AWS run rough shod over them. When I was at AWS ProServe, we didn’t even bother coming up with talki…

[deleted]

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#128

Post by slickytail »

torginus wrote: post #101
potatolicious wrote: post #88

Earlier quoted context omitted.

NAND is very, very slow relative to RAM, so you'd pay a huge performance penalty there. But maybe more importantly my impression is that memory contents mutate pretty heavily during inference (you're not just storing the fixed weights), so I'd be pretty concerned about NAND wear. Mutating a single bit on a NAND chip a million times over just results in a large pile of dead NAND chips.

No it's not slow - a single NAND chip in SSDs offers >1GB of bandwidth - inside the chip there are 100+ wafers actually holding the data, but in SSDs only one of them is active when reading/writing. You could probably make special NAND chips where all of them can be active at the same time, which means you could get 100GB+ bandwidth out of a single chip. This would be useless for data storage scenarios, but very usef…

The memory bandwidth on an H100 is 3TB/s, for reference. This number is the limiting factor in the size of modern LLMs. 100GB/s isn't even in the realm of viability.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#129

Post by vlovich123 »

1. They have many machines to split the load over 2. MoE architecture that lets them shard experts across different machines - 1 machine handles generating 1 token of context before the entire thing is shipped off to the next expert for the next token. This reduces bandwidth requirements by 1/N as well as the amount of VRAM needed on any single machine 3. They batch tokens from multiple users to further reduce memory bandwidth (eg they compute the math for some given weights on multiple users). This reduces bandwidth requirements significantly as well.

So basically the main tricks are batching (only relevant when you have > 1 query to process) and MoE sharding.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#130

Post by JoshuaDavid »

scarface_74 wrote: post #111
baxtr wrote: post #91

Earlier quoted context omitted.

The only one who can stop Google is Google. They’ll definitely have the best model, but there is a chance they will f*up the product / integration into their products.

It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP. But then again even there, their reputation for abandoning products, lack of customer service, condescension when it came to large enterprises’ “legacy tech” lets Microsoft who is king of hand holding big enterprise and even AWS run rough shod over them. When I was at AWS ProServe, we didn’t even bother coming up with talki…

Google employees collectively have a lot of talent.
Post reply on HN