Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT could cost over $700k per day to operate

businessinsider.com

Post reply on HN
Print view View original

101–110 of 223 posts

Re: ChatGPT could cost over $700k per day to operate

#101

Post by reaperman »

zamnos wrote: post #43
vitus wrote: post #21

Earlier quoted context omitted.

The estimate in the article pins most of the cost on inference, not training, so diurnal patterns are unfortunately not as useful here. > While training ChatGPT's large language models likely costs tens of millions of dollars, operational expenses, or inference costs, "far exceed training costs when deploying a model at any reasonable scale," Patel and Afzal Ahmad, another analyst at SemiAnalysis, told Forbes. "In fa…

Why wouldn't inference follow diurnal patterns?

Agreed. That seems backwards. Training would not follow circadian rhythms, inference would.

Re: ChatGPT could cost over $700k per day to operate

#102

Post by CuriouslyC »

marginalia_nu wrote: post #54
CuriouslyC wrote: post #48

Earlier quoted context omitted.

There are a few avenues. Further specialization of hardware around LLMs, better quantization (3 bits/p seems promising), improved attention mechanisms, use of distilled models for common prompts, etc.

This would be optimizations, which is not really the same thing as moore's law-like growth which was absolutely mind-boggling, like it's hard to even wrap your head around how fast tech was moving in that period since humans don't really grok exponentials too well, we just think they look like second degree polynomials.

Probabilistic computing offers the potential of a return to that pace of progress. We spend a lot of silicon on squashing things to 0/1 with error correction, but using analog voltages to carry information and relying on parameter redundancy for error correction could lead to much greater efficiency both in terms of OPS/mm^2 and OPS/watt.

Re: ChatGPT could cost over $700k per day to operate

#103

Post by coldpie »

consp wrote: post #64
javajosh wrote: post #52

Earlier quoted context omitted.

It's almost like ChatGPT is nerd-sniping us all. But we like it and it's useful to us.

Don't generalize, I've never used it and never seen any reason to use it.

You should give it a shot, keep an open mind. Yes, it will get things wrong and yes, it will disappoint you. But every conversation I've had with it holds some surprises for me. It's a tool you'll need to learn to use before it actually becomes useful, but give it a couple minutes a few times a week and you'll start to see the diamonds in the rough.

Re: ChatGPT could cost over $700k per day to operate

#104

Post by cft »

m3kw9 wrote: post #91
mnky9800n wrote: post #67

Earlier quoted context omitted.

I paid for it, then stopped paying for it. It seemed to be completely novel and fun to use but after a while it just became a pain. I tried writing a library to act as a code assistant for data science but trying to get it to write the codes that were useful to me took around the same amount of time as writing them myself. What you say is completely correct, it gets to around 70% of whatever you ask it to do then you…

You can use bing chat for free. Supposedly powered by gpt4.

Bing gives you much shorter answers, often without code examples

Re: ChatGPT could cost over $700k per day to operate

#105

Post by [deleted] »

zerop wrote: post #8
crop_rotation wrote: post #5

Isn't Microsoft too late to the chip game? Both their competitors AWS and GCP have their own ML training chips, while Azure has nothing. The article seems like they may have something by next year, but by that time the competition would have evolved even more. Nadella has executed on strategy brilliantly but on the cloud seems it seems like the tech innovation is a bit lagging. It hasn't mattered that much so far tho…

Microsoft reportedly working on its own AI chips that may rival Nvidia’s https://www.theverge.com/2023/4/18/23687912/microsoft-athena...

[deleted]

Re: ChatGPT could cost over $700k per day to operate

#107

Post by mym1990 »

octopoc wrote: post #71
jermaustin1 wrote: post #30

Earlier quoted context omitted.

I have started incorporating ChatGPT into my side project. I'm not integrating it into software or anything like that, but using it as a tool for product ideas, product photo ideas, funny social media captions, figuring out the best hashtags to use, product tagging and categorization. It has proven to be a great way to get everything to around 70%, then send off to my assistant for the remaining 30% of polish. So at…

I've been using it for product ideas as well. Occasionally it's brilliant but sometimes it's so bad it's funny, like when it suggested Seven-Year Itch for a perfume brand.

Don't they say something about seemingly brilliant ideas being too obvious or usually not working out, and the terrible ones being the ones that are have more potential to be diamonds in the rough?

Re: ChatGPT could cost over $700k per day to operate

#108

Post by bemmu »

kylecazar wrote: post #90
BlackSwanMan wrote: post #81

Earlier quoted context omitted.

The cheap version does all of that much worse. I don't understand how people can be so cheap.

There are so many subscription products these days that it makes sense to assess the value of each carefully before committing. If you're like me you tend to forget you're paying for something, and have to do a yearly purge. It's very easy to slowly but surely rack up hundreds of dollars a month in subscription services.

My habit is to cancel right away when I start a subscription. You can still use the service for the month you paid for, and it's easy to restart it if you do still find yourself attempting to use it a month later.

Re: ChatGPT could cost over $700k per day to operate

#109

Post by nerdix »

gabereiser wrote: post #40
capableweb wrote: post #20

Earlier quoted context omitted.

Considering that Microsoft is a huge investor in OpenAI, I'd be surprised they pay anything at all in reality.

that's ridiculous, OpenAI is paying. Granted Microsoft invested heavily into OpenAI but those are two separate financial transactions. Sure you can rationalize in your head that IN-OUT=DIFF but that's not how books are kept.

Microsoft invested $1 billion OpenAI in 2019 and half of that amount was in Azure credits.

I'm not sure about the most recent $10 billion investment but I wouldn't be surprised if a significant amount of it is in Azure credits as well.

While that's not "free" (they exchanged equity for it), it's likely not an expense (or at least not an expense that they have to cover fully).

Re: ChatGPT could cost over $700k per day to operate

#110

Post by redmaverick »

mnky9800n wrote: post #67
jermaustin1 wrote: post #30

Earlier quoted context omitted.

I have started incorporating ChatGPT into my side project. I'm not integrating it into software or anything like that, but using it as a tool for product ideas, product photo ideas, funny social media captions, figuring out the best hashtags to use, product tagging and categorization. It has proven to be a great way to get everything to around 70%, then send off to my assistant for the remaining 30% of polish. So at…

I paid for it, then stopped paying for it. It seemed to be completely novel and fun to use but after a while it just became a pain. I tried writing a library to act as a code assistant for data science but trying to get it to write the codes that were useful to me took around the same amount of time as writing them myself. What you say is completely correct, it gets to around 70% of whatever you ask it to do then you…

free version is an order of magnitude worse than the paid version. It is still good though.
Post reply on HN