Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT could cost over $700k per day to operate

businessinsider.com

Post reply on HN
Print view View original

51–60 of 223 posts

Re: ChatGPT could cost over $700k per day to operate

#51

Post by pluijzer »

pdksam wrote: post #12
Dave3of5 wrote: post #9

Interesting numbers. That roughly equates to about $250 million per year plus I don't know how much training is costing them to keep the model up to date and suchlike. The company also has about 375 employees. I've no idea how much they get paid but I used $200k as a yearly cost and that comes to $75 million. That's about 3:1 cost of operating the services to paying employees. That seems quite high as I've never been…

200k is too small, strong sde1s at Amazon get paid that much in hcol areas. Closer to 500k.

As a side, I am a bit shocked by these numbers. Is this an American thing? I understand myself to be good software engineer with good well rounded experience of 14+ years. Yet my income, in Europe, is really above 100k.

What I am wondering, for those earning 500k, how big is your work load/stress. Would this be a 9-5 job you leave at the office when going home. Or does a job that earns so much consume your life?

Re: ChatGPT could cost over $700k per day to operate

#52

Post by javajosh »

jermaustin1 wrote: post #30
pc_edwin wrote: post #15

This might sound tone deaf but thats seems extremely cheap to me. It means the cost range for chatGPT is between $300m - $500m per year. If the 100m users is accurate, it means they only need to convert low single digit percentage to paying users to break even. Now it makes sense why they chose to charge $20/m, I predicted much higher.

I have started incorporating ChatGPT into my side project. I'm not integrating it into software or anything like that, but using it as a tool for product ideas, product photo ideas, funny social media captions, figuring out the best hashtags to use, product tagging and categorization. It has proven to be a great way to get everything to around 70%, then send off to my assistant for the remaining 30% of polish. So at…

It's almost like ChatGPT is nerd-sniping us all. But we like it and it's useful to us.

Re: ChatGPT could cost over $700k per day to operate

#54

Post by marginalia_nu »

CuriouslyC wrote: post #48
marginalia_nu wrote: post #26

Earlier quoted context omitted.

But what is being optimized? Hardware sure isn't getting faster in a hurry, and I don't see anything on the horizon that will aid in optimizing software.

There are a few avenues. Further specialization of hardware around LLMs, better quantization (3 bits/p seems promising), improved attention mechanisms, use of distilled models for common prompts, etc.

This would be optimizations, which is not really the same thing as moore's law-like growth which was absolutely mind-boggling, like it's hard to even wrap your head around how fast tech was moving in that period since humans don't really grok exponentials too well, we just think they look like second degree polynomials.

Re: ChatGPT could cost over $700k per day to operate

#55

Post by reitanqild »

marginalia_nu wrote: post #42
ben_w wrote: post #34

Earlier quoted context omitted.

The various open source LLMs are doing things like reducing bits-per-parameter to reduce hardware requirements; if they're using COTS hardware it almost certainly isn't optimised for their specific models; Moore's Law is pretty heavily reinterpreted, so although we normally care about "operations per second at a fixed number of monies" what matters here is "joules per operation" which can improve a by a huge margin e…

Moore's law was an effect that stemmed from the locally exponential efficiency increase from designing computers using computers, each iteration growing more powerful and capable of designing still more powerful hardware. 10% here and there is very small compared to the literal orders magnitude improvements during the reign of Moore's Law. I don't really see anything like that here.

> 10% here and there is very small compared to the literal orders magnitude improvements during the reign of Moore's Law.

I can't confirm it, but I noticed this comment says "gpu tech has beat Moore’s law for DNNs the last several years":

https://news.ycombinator.com/item?id=35653231

Re: ChatGPT could cost over $700k per day to operate

#56

Post by jongjong »

pohl wrote: post #37
qwytw wrote: post #18

Earlier quoted context omitted.

> these days Wasn't it always more or less like this?

No.

Agreed. I entered the workforce as a developer in 2012; it felt like the tail-end of the tech boom but intelligence was still valued. Nowadays, most people don't even recognize intelligence, let alone value it.

Re: ChatGPT could cost over $700k per day to operate

#57

Post by barkerja »

ren_engineer wrote: post #46
pc_edwin wrote: post #15

This might sound tone deaf but thats seems extremely cheap to me. It means the cost range for chatGPT is between $300m - $500m per year. If the 100m users is accurate, it means they only need to convert low single digit percentage to paying users to break even. Now it makes sense why they chose to charge $20/m, I predicted much higher.

interesting that the article mentions Microsoft is trying to build their own AI chips to help reduce cost, they are years behind Google in that effort. Google still has the advantage in both research and hardware, they just fumbled the execution but it's still very early in the game

> they are years behind Google in that effort. Google still has the advantage in both research and hardware, they just fumbled the execution but it's still very early in the game

Where do we think Apple is in regards to this? If anyone has the upper-hand here with hardware, I would think it'd be Apple. But there's been zero indication that Apple has been working on any sort of generative AI.

Re: ChatGPT could cost over $700k per day to operate

#58

Post by behnamoh »

ren_engineer wrote: post #46
pc_edwin wrote: post #15

This might sound tone deaf but thats seems extremely cheap to me. It means the cost range for chatGPT is between $300m - $500m per year. If the 100m users is accurate, it means they only need to convert low single digit percentage to paying users to break even. Now it makes sense why they chose to charge $20/m, I predicted much higher.

interesting that the article mentions Microsoft is trying to build their own AI chips to help reduce cost, they are years behind Google in that effort. Google still has the advantage in both research and hardware, they just fumbled the execution but it's still very early in the game

The past 3 years showed that Google is not as ahead as people thought they were.

Re: ChatGPT could cost over $700k per day to operate

#59

Post by marginalia_nu »

reitanqild wrote: post #55
marginalia_nu wrote: post #42

Earlier quoted context omitted.

Moore's law was an effect that stemmed from the locally exponential efficiency increase from designing computers using computers, each iteration growing more powerful and capable of designing still more powerful hardware. 10% here and there is very small compared to the literal orders magnitude improvements during the reign of Moore's Law. I don't really see anything like that here.

> 10% here and there is very small compared to the literal orders magnitude improvements during the reign of Moore's Law. I can't confirm it, but I noticed this comment says "gpu tech has beat Moore’s law for DNNs the last several years": https://news.ycombinator.com/item?id=35653231

We're actually at an inflection point where this isn't the case anymore.

For a long time, GPU hardware basically became more powerful with each generation, but prices stayed roughly the same plus minus inflation. Last couple of years, this trend has broken. You pay double or even quadruple the price for a relatively tenuous increase in performance.

Re: ChatGPT could cost over $700k per day to operate

#60

Post by brrrrrm »

ren_engineer wrote: post #46
pc_edwin wrote: post #15

This might sound tone deaf but thats seems extremely cheap to me. It means the cost range for chatGPT is between $300m - $500m per year. If the 100m users is accurate, it means they only need to convert low single digit percentage to paying users to break even. Now it makes sense why they chose to charge $20/m, I predicted much higher.

interesting that the article mentions Microsoft is trying to build their own AI chips to help reduce cost, they are years behind Google in that effort. Google still has the advantage in both research and hardware, they just fumbled the execution but it's still very early in the game

In terms of production pipeline, sure. But since there’s an established model architecture, the hardware doesn’t need to be super generic. Google had to build TPUs in a world where everything was still rapidly changing - deciding on things like precision, memory bandwidth, SRAM size etc. Microsoft could theoretically stamp out some GPT ASICS and call it a day.
Post reply on HN