Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#111

Post by scarface_74 »

baxtr wrote: post #91
ActorNightly wrote: post #28

Earlier quoted context omitted.

Yes. Google is probably gonna win the LLM game tbh. They had a massive head start with TPUs which are very energy efficient compared to Nvidia Cards.

The only one who can stop Google is Google. They’ll definitely have the best model, but there is a chance they will f*up the product / integration into their products.

It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP.

But then again even there, their reputation for abandoning products, lack of customer service, condescension when it came to large enterprises’ “legacy tech” lets Microsoft who is king of hand holding big enterprise and even AWS run rough shod over them.

When I was at AWS ProServe, we didn’t even bother coming up with talking points when competing with GCP except to point out how they abandon services. Was it partially FUD? Probably. But it worked.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#112

Post by scarface_74 »

airhangerf15 wrote: post #11

An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper" I don't think people realize the size of these compute u…

This isn’t like how Google was able to buy up dark fiber cheaply and use it.

From what I understand, this hardware has a high failure rate over the long term especially because of the heat they generate.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#113

Post by andy99 »

eulgro wrote: post #106
semi-extrinsic wrote: post #38

Earlier quoted context omitted.

What do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.

> 13 000 BTU/hr In sane units: 3.8 kW

You mean 1.083 tons of refrigeration

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#114

Post by DaiPlusPlus »

jonplackett wrote: post #107
mattmanser wrote: post #62

Earlier quoted context omitted.

Someone's take on AI was that we're collectively investing billions in data centers that will be utterly worthless in 10 years. Unlike the investments in railways or telephone cables or roads or any other sort of architecture, this investment has a very short lifespan. Their point was that whatever your take on AI, the present investment in data centres is a ridiculous waste and will always end up as a huge net loss…

They probably are right, but a counter argument could be how people thought going to the moon was pointless and insanely expensive, but the technology to put stuff in space and have GPS and comms satellites probably paid that back 100x

I don’t mean to invalidate your point (about genuine value arising from innovations originating from the Apollo program), but GPS and comms satellites (and heck, the Internet) are all products of nuclear weapons programs rather than civilian space exploration programs (ditto the Space Shuttle, and I could go on…).

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#115

Post by eitally »

airhangerf15 wrote: post #11

An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper" I don't think people realize the size of these compute u…

What I wonder is what this means for Coreweave, Lambda and the rest, who are essentially just renting out fleets of racks like this. Does it ultimately result in acquisition by a larger player? Severe loss of demand? Can they even sell enough to cover the capex costs?

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#116

Post by hdgvhicv »

ksherlock wrote: post #90
semi-extrinsic wrote: post #38

Earlier quoted context omitted.

What do you mean 10 years? You can pick up a DGX-1 on Ebay right now for less than $10k. 256 GB vRAM (HBM2 nonetheless), NVLink capability, 512 GB RAM, 40 CPU cores, 8 TB SSD, 100 Gbit HBAs. Equivalent non-Nvidia branded machines are around $6k. They are heavy, noisy like you would not believe, and a single one just about maxes out a 16A 240V circuit. Which also means it produces 13 000 BTU/hr of waste heat.

It's not waste heat if you only run it in the winter.

Opt if you ignore that both gas furnaces and heat pumps are more efficient than resistive loads.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#117

Post by klik99 »

ActorNightly wrote: post #97
fakedang wrote: post #59

Earlier quoted context omitted.

Yeah honestly. They could just try selling solutions and SLAs combining their TPU hardware with on-prem SOTA models and practically dominate enterprise. From what I understand, that's GCP's gameplay too for most regulated enterprise clients.

Googles bread and butter is advertising, so they have a huge interest in keeping things in house. Data is more valuable to them than money from hardware sales. Even then, I think that their primary use case is going to be consumer grade good AI on phones. I dunno why Gemma QAT model fly so low on the radar, but you can basically get full scale Llamma 3 like performance from a single 3090 now, at home.

It’s my understanding that google makes bulk of ad money from search ads - sure they harvest a ton of data but it isn’t as valuable to them as you’d think. I suspect they know that could change so they’re hoovering up as much as they can to hedge their bets. Meta on the other hand is all about targeted ads.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#118

Post by adw »

eitally wrote: post #115
airhangerf15 wrote: post #11

An H100 is a $20k USD card and has 80GB of vRAM. Imagine a 2U rack server with $100k of these cards in it. Now imagine an entire rack of these things, plus all the other components (CPUs, RAM, passive cooling or water cooling) and you're talking $1 million per rack, not including the costs to run them or the engineers needed to maintain them. Even the "cheaper" I don't think people realize the size of these compute u…

What I wonder is what this means for Coreweave, Lambda and the rest, who are essentially just renting out fleets of racks like this. Does it ultimately result in acquisition by a larger player? Severe loss of demand? Can they even sell enough to cover the capex costs?

These are also depreciating assets.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#119

Post by vl »

jonplackett wrote: post #107
mattmanser wrote: post #62

Earlier quoted context omitted.

Someone's take on AI was that we're collectively investing billions in data centers that will be utterly worthless in 10 years. Unlike the investments in railways or telephone cables or roads or any other sort of architecture, this investment has a very short lifespan. Their point was that whatever your take on AI, the present investment in data centres is a ridiculous waste and will always end up as a huge net loss…

They probably are right, but a counter argument could be how people thought going to the moon was pointless and insanely expensive, but the technology to put stuff in space and have GPS and comms satellites probably paid that back 100x

Reality is that we don’t know how much of a trope this statement is.

I think we would get all this technology without going to the moon or Space Shuttle program. GPS, for example, was developed for military applications initially.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#120

Post by minimaxir »

v5v3 wrote: post #34
minimaxir wrote: post #2

> Sure, they have huge GPU clusters That's a really, really big "sure." Almost every trick to run a LLM at OpenAI's scale is a trade secret and may not be easily understood by mere mortals anyways (e.g. bare-metal CUDA optimizations)

Trade secret? With all the staff poaching the trade secrets may have now leaked?

That's half the reason tech companies poach.
Post reply on HN