Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#131

Post by bespokedevelopr »

mattmanser wrote: post #62
invaliduser wrote: post #22

Earlier quoted context omitted.

Even is the AI bubble does not pops, your prediction about those servers being available on ebay in 10 years will likely be true, because some datacenters will simply upgrade their hardware and resell their old ones to third parties.

Someone's take on AI was that we're collectively investing billions in data centers that will be utterly worthless in 10 years. Unlike the investments in railways or telephone cables or roads or any other sort of architecture, this investment has a very short lifespan. Their point was that whatever your take on AI, the present investment in data centres is a ridiculous waste and will always end up as a huge net loss…

If it is all a waste and a bubble, I wonder what the long term impact will be of the infrastructure upgrades around these dcs. A lot of new HV wires and substations are being built out. Cities are expanding around clusters of dcs. Are they setting themselves up for a new rust belt?

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#132

Post by worik »

I do not have a technical answer, but I have the feeling that the concept of "loss leaders" is useful

IMO outfits like OpenAI are burning metric shit tonnes of cash serving these models. It pails in comparison to the mega shit tonnes of cash used to train the models.

They hope to gain market share before they start charging customers what it costs.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#134

Post by stogot »

ActorNightly wrote: post #28
tough wrote: post #17

Earlier quoted context omitted.

Doesn't google have TPU's that makes inference of their own models much more profitable than say having to rent out NVDIA cards? Doesn't OpenAI depend mostly on its relationship/partnership with Microsoft to get GPUs to inference on? Thanks for the links, interesting book!

Yes. Google is probably gonna win the LLM game tbh. They had a massive head start with TPUs which are very energy efficient compared to Nvidia Cards.

Hasn’t the Inferentia chip been around long enough to make the same argument? AWS and Google probably have the same order of magnitude of their own custom chips

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#135

Post by abeyer »

bespokedevelopr wrote: post #131
mattmanser wrote: post #62

Earlier quoted context omitted.

Someone's take on AI was that we're collectively investing billions in data centers that will be utterly worthless in 10 years. Unlike the investments in railways or telephone cables or roads or any other sort of architecture, this investment has a very short lifespan. Their point was that whatever your take on AI, the present investment in data centres is a ridiculous waste and will always end up as a huge net loss…

If it is all a waste and a bubble, I wonder what the long term impact will be of the infrastructure upgrades around these dcs. A lot of new HV wires and substations are being built out. Cities are expanding around clusters of dcs. Are they setting themselves up for a new rust belt?

Or early provisioning for massively expanded electric transit and EV charging infrastructure, perhaps.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#136

Post by meatmanek »

tgma wrote: post #124
hdgvhicv wrote: post #116

Earlier quoted context omitted.

Opt if you ignore that both gas furnaces and heat pumps are more efficient than resistive loads.

Heat pump sure, but how is gas furnace more efficient than resistive load inside the house? Do you mean more economical rather than more efficient (due to gas being much cheaper/unit of energy)?

Depends where your electricity comes from. If you're burning fossil fuels to make electricity, that's only about 40% efficient, so you need to burn 2.5x as much fuel to get the same amount of heat into the house.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#137

Post by blibble »

canyon289 wrote: post #6

I work at Google on these systems everyday (caveat this is my own words not my employers)). So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that. However I can share this written by my colleagues! You'll find great explanations about accelerator architectures and the considerations made to make things fast. https://jax-ml.git…

> So I simultaneously can tell you that its smart people really thinking about every facet of the problem, and I can't tell you much more than that.

"we do 1970s mainframe style timesharing"

there, that was easy

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#138

Post by serf »

scarface_74 wrote: post #111
baxtr wrote: post #91

Earlier quoted context omitted.

The only one who can stop Google is Google. They’ll definitely have the best model, but there is a chance they will f*up the product / integration into their products.

It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP. But then again even there, their reputation for abandoning products, lack of customer service, condescension when it came to large enterprises’ “legacy tech” lets Microsoft who is king of hand holding big enterprise and even AWS run rough shod over them. When I was at AWS ProServe, we didn’t even bother coming up with talki…

>It would take talent for them to mess up hosting businesses who want to use their TPUs on GCP.

there are few groups as talented at losing a head start as google.

Post reply on HN