Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT could cost over $700k per day to operate

businessinsider.com

Post reply on HN
Print view View original

41–50 of 223 posts

Re: ChatGPT could cost over $700k per day to operate

#41

Post by nathancahill »

wouldbecouldbe wrote: post #38
pc_edwin wrote: post #15

This might sound tone deaf but thats seems extremely cheap to me. It means the cost range for chatGPT is between $300m - $500m per year. If the 100m users is accurate, it means they only need to convert low single digit percentage to paying users to break even. Now it makes sense why they chose to charge $20/m, I predicted much higher.

That's just server costs, not labor hours for support, management & innovation.

Yeah think of all the labor hours, how many people do they have back there typing up the answers?

Re: ChatGPT could cost over $700k per day to operate

#42

Post by marginalia_nu »

ben_w wrote: post #34
marginalia_nu wrote: post #26

Earlier quoted context omitted.

But what is being optimized? Hardware sure isn't getting faster in a hurry, and I don't see anything on the horizon that will aid in optimizing software.

The various open source LLMs are doing things like reducing bits-per-parameter to reduce hardware requirements; if they're using COTS hardware it almost certainly isn't optimised for their specific models; Moore's Law is pretty heavily reinterpreted, so although we normally care about "operations per second at a fixed number of monies" what matters here is "joules per operation" which can improve a by a huge margin e…

Moore's law was an effect that stemmed from the locally exponential efficiency increase from designing computers using computers, each iteration growing more powerful and capable of designing still more powerful hardware.

10% here and there is very small compared to the literal orders magnitude improvements during the reign of Moore's Law.

I don't really see anything like that here.

Re: ChatGPT could cost over $700k per day to operate

#43

Post by zamnos »

vitus wrote: post #21
zamnos wrote: post #16

Earlier quoted context omitted.

They're interesting numbers, but the linked article's cite amounts to: > ChatGPT could cost OpenAI up to $700,000 a day to run due to "expensive servers," an analyst told The Information. which, pardon me, but no shit. Before I break out my back of the envelope calculator, on how many biggest GPU instances in Azure that is, the real question is what their underlying assumptions are, and where they're getting them fro…

The estimate in the article pins most of the cost on inference, not training, so diurnal patterns are unfortunately not as useful here. > While training ChatGPT's large language models likely costs tens of millions of dollars, operational expenses, or inference costs, "far exceed training costs when deploying a model at any reasonable scale," Patel and Afzal Ahmad, another analyst at SemiAnalysis, told Forbes. "In fa…

Why wouldn't inference follow diurnal patterns?

Re: ChatGPT could cost over $700k per day to operate

#44

Post by lumost »

MuffinFlavored wrote: post #36
pc_edwin wrote: post #15

This might sound tone deaf but thats seems extremely cheap to me. It means the cost range for chatGPT is between $300m - $500m per year. If the 100m users is accurate, it means they only need to convert low single digit percentage to paying users to break even. Now it makes sense why they chose to charge $20/m, I predicted much higher.

What does it cost to run per day versus what did it cost to train?

These costs are falling by factors of 2 every year based solely on hardware progress ( gpu tech has beat Moore’s law for DNNs the last several years). ChatGPT may become much cheaper to operate/train in the future.

Re: ChatGPT could cost over $700k per day to operate

#45

Post by [deleted] »

xyzzy123 wrote: post #35
crop_rotation wrote: post #5

Isn't Microsoft too late to the chip game? Both their competitors AWS and GCP have their own ML training chips, while Azure has nothing. The article seems like they may have something by next year, but by that time the competition would have evolved even more. Nadella has executed on strategy brilliantly but on the cloud seems it seems like the tech innovation is a bit lagging. It hasn't mattered that much so far tho…

Microsoft have a pretty considerable advantage - a valuable production workload to optimise and strong knowledge of what that looks like. Sometimes it's better to be the fast follower.

[deleted]

Re: ChatGPT could cost over $700k per day to operate

#46

Post by ren_engineer »

pc_edwin wrote: post #15

This might sound tone deaf but thats seems extremely cheap to me. It means the cost range for chatGPT is between $300m - $500m per year. If the 100m users is accurate, it means they only need to convert low single digit percentage to paying users to break even. Now it makes sense why they chose to charge $20/m, I predicted much higher.

interesting that the article mentions Microsoft is trying to build their own AI chips to help reduce cost, they are years behind Google in that effort. Google still has the advantage in both research and hardware, they just fumbled the execution but it's still very early in the game

Re: ChatGPT could cost over $700k per day to operate

#47

Post by robertlagrant »

sebzim4500 wrote: post #32
crop_rotation wrote: post #5

Isn't Microsoft too late to the chip game? Both their competitors AWS and GCP have their own ML training chips, while Azure has nothing. The article seems like they may have something by next year, but by that time the competition would have evolved even more. Nadella has executed on strategy brilliantly but on the cloud seems it seems like the tech innovation is a bit lagging. It hasn't mattered that much so far tho…

It feels like Microsoft realized that sales/support matters way more than technology when it comes to cloud services. See how Azure has a ton of users despite being much worse than AWS or GCP by every objective metric. I would imagine that Google's reputation as a company where it is impossible to ever talk to a human (even when you have a 7 figure annual spend) hurts them in this space.

I think it's hard to tell what are real cloud users vs what are Office 365 etc users in their reporting.

Re: ChatGPT could cost over $700k per day to operate

#48

Post by CuriouslyC »

marginalia_nu wrote: post #26
Damogran6 wrote: post #23

Earlier quoted context omitted.

Moore's law-ish like optimization. You Z80 computer cost $700 in the lat 70's...they're now in sub-$1 embedded controllers.

But what is being optimized? Hardware sure isn't getting faster in a hurry, and I don't see anything on the horizon that will aid in optimizing software.

There are a few avenues. Further specialization of hardware around LLMs, better quantization (3 bits/p seems promising), improved attention mechanisms, use of distilled models for common prompts, etc.

Re: ChatGPT could cost over $700k per day to operate

#49

Post by warent »

nathancahill wrote: post #41
wouldbecouldbe wrote: post #38

Earlier quoted context omitted.

That's just server costs, not labor hours for support, management & innovation.

Yeah think of all the labor hours, how many people do they have back there typing up the answers?

And who pays the tiny people in my magic light box to draw the answers on the window?!
Post reply on HN