Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 379 posts

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#181

Post by nucleardog »

potatolicious wrote: post #76
invaliduser wrote: post #22

Earlier quoted context omitted.

Even is the AI bubble does not pops, your prediction about those servers being available on ebay in 10 years will likely be true, because some datacenters will simply upgrade their hardware and resell their old ones to third parties.

Would anybody buy the hardware though? Sure, datacenters will get rid of the hardware - but only because it's no longer commercially profitable run them, presumably because compute demands have eclipsed their abilities. It's kind of like buying a used GeForce 980Ti in 2025. Would anyone buy them and run them besides out of nostalgia or curiosity? Just the power draw makes them uneconomical to run. Much more likely ev…

> Sure, datacenters will get rid of the hardware - but only because it's no longer commercially profitable run them, presumably because compute demands have eclipsed their abilities.

I think the existence of a pretty large secondary market for enterprise servers and such kind of shows that this won't be the case.

Sure, if you're AWS and what you're selling _is_ raw compute, then couple generation old hardware may not be sufficiently profitable for you anymore... but there are a lot of other places that hardware could be applied to with different requirements or higher margins where it may still be.

Even if they're only running models a generation or two out of date, there are a lot of use cases today, with today's models, that will continue to work fine going forward.

And that's assuming it doesn't get replaced for some other reason that only applies when you're trying to sell compute at scale. A small uptick in the failure rate may make a big dent at OpenAI but not for a company that's only running 8 cards in a rack somewhere and has a few spares on hand. A small increase in energy efficiency might offset the capital outlay to upgrade at OpenAI, but not for the company that's only running 8 cards.

I think there's still plenty of room in the market in places where running inference "at cost" would be profitable that are largely untapped right now because we haven't had a bunch of this hardware hit the market at a lower cost yet.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#182

Post by AdieuToLogic »

catigula wrote: post #157
YossarianFrPrez wrote: post #154

Earlier quoted context omitted.

Can you explain what you mean about 'not needing to be solved'? There are versions of that kind of critique that would seem, at least on the surface, to better apply to finance or flash trading. I ask because scaling an system that a substantially chunk of the population finds incredibly useful, including for the more efficient production of public goods (scientific research, for example) does seem like a problem tha…

[flagged]

> People are starving to death and the world's brightest engineers are ...

This is a political will, empathy, and leadership problem. Not an engineering problem.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#183

Post by seneca »

catigula wrote: post #157
YossarianFrPrez wrote: post #154

Earlier quoted context omitted.

Can you explain what you mean about 'not needing to be solved'? There are versions of that kind of critique that would seem, at least on the surface, to better apply to finance or flash trading. I ask because scaling an system that a substantially chunk of the population finds incredibly useful, including for the more efficient production of public goods (scientific research, for example) does seem like a problem tha…

[flagged]

Famine in the modern world is almost entirely caused by dysfunctional governments and/or armed conflicts. Engineers have basically nothing to do with either of those.

This sort of "there are bad things in the world, therefore focusing on anything else is bad" thinking is generally misguided.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#184

Post by trhway »

catigula wrote: post #157
YossarianFrPrez wrote: post #154

Earlier quoted context omitted.

Can you explain what you mean about 'not needing to be solved'? There are versions of that kind of critique that would seem, at least on the surface, to better apply to finance or flash trading. I ask because scaling an system that a substantially chunk of the population finds incredibly useful, including for the more efficient production of public goods (scientific research, for example) does seem like a problem tha…

[flagged]

the existence of poor hungry people feeds the fear of becoming poor and hungry which drives those brightest engineers. I.e. the things work as intended, unfortunately.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#185

Post by Invictus0 »

93po wrote: post #155
abathologist wrote: post #63

One clever ingredient in OpenAI's secret sauce is billions of dollars of losses. About $5 billion dollars lost in 2024. https://www.cnbc.com/2024/09/27/openai-sees-5-billion-loss-t...

they would be break-even if all they did was serve existing models and got rid of everything related to R&D

An AI lab with no R&D. Truly a hacker news moment

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#186

Post by AdieuToLogic »

hattmall wrote: post #164
catigula wrote: post #157

Earlier quoted context omitted.

[flagged]

The only solution to those people starving to death is to kill the people that benefit from them starving to death. It's a solved problem, the solution isn't palatable. No one is starving to death because of a lack of engineering prowess.

>> People are starving to death ...

> The only solution to those people starving to death is to kill the people that benefit from them starving to death.

There are solutions other than "to kill the people that benefit", such as what have existed for many years, including but not limited to:

  - Efforts such as the recently emasculated USAID[0].
  - Humanitarian NGO's[1] such as the World Central Kitchen[2]
    and the Red Cross[3].
  - The will of those who could help to help those in need[4].
Note that none of the aforementioned require executions nor engineering prowess.

0 - https://en.wikipedia.org/wiki/United_States_Agency_for_Inter...

1 - https://en.wikipedia.org/wiki/Non-governmental_organization

2 - https://wck.org/

3 - https://en.wikipedia.org/wiki/International_Red_Cross_and_Re...

4 - https://en.wikipedia.org/wiki/Empathy

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#187

Post by venusenvy47 »

The big players use parallel processing of multiple users to keep the GPUs and memory filled as much as possible during the inference they are providing to users. They can make use of the fact that they have a fairly steady stream of requests coming into their data centers at all times. This article describes some of how this is accomplished.

https://www.infracloud.io/blogs/inference-parallelism/

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#189

Post by darth_avocado »

seneca wrote: post #183
catigula wrote: post #157

Earlier quoted context omitted.

[flagged]

Famine in the modern world is almost entirely caused by dysfunctional governments and/or armed conflicts. Engineers have basically nothing to do with either of those. This sort of "there are bad things in the world, therefore focusing on anything else is bad" thinking is generally misguided.

Famine is mostly political but engineers (not all of them) definitely have to do with it. If you’re building powerful AI for corporations that are then involved with the political entities that caused the famine, then you can’t claim to basically have nothing to do with it.

Re: Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?

#190

Post by shigawire »

AdieuToLogic wrote: post #182
catigula wrote: post #157

Earlier quoted context omitted.

[flagged]

> People are starving to death and the world's brightest engineers are ... This is a political will, empathy, and leadership problem. Not an engineering problem.

Those problems might be more tractable if all of our best and brightest were working on them.
Post reply on HN