Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT could cost over $700k per day to operate

businessinsider.com

Post reply on HN
Print view View original

91–100 of 223 posts

Re: ChatGPT could cost over $700k per day to operate

#91

Post by m3kw9 »

mnky9800n wrote: post #67
jermaustin1 wrote: post #30

Earlier quoted context omitted.

I have started incorporating ChatGPT into my side project. I'm not integrating it into software or anything like that, but using it as a tool for product ideas, product photo ideas, funny social media captions, figuring out the best hashtags to use, product tagging and categorization. It has proven to be a great way to get everything to around 70%, then send off to my assistant for the remaining 30% of polish. So at…

I paid for it, then stopped paying for it. It seemed to be completely novel and fun to use but after a while it just became a pain. I tried writing a library to act as a code assistant for data science but trying to get it to write the codes that were useful to me took around the same amount of time as writing them myself. What you say is completely correct, it gets to around 70% of whatever you ask it to do then you…

You can use bing chat for free. Supposedly powered by gpt4.

Re: ChatGPT could cost over $700k per day to operate

#92

Post by Damogran6 »

marginalia_nu wrote: post #59
reitanqild wrote: post #55

Earlier quoted context omitted.

> 10% here and there is very small compared to the literal orders magnitude improvements during the reign of Moore's Law. I can't confirm it, but I noticed this comment says "gpu tech has beat Moore’s law for DNNs the last several years": https://news.ycombinator.com/item?id=35653231

We're actually at an inflection point where this isn't the case anymore. For a long time, GPU hardware basically became more powerful with each generation, but prices stayed roughly the same plus minus inflation. Last couple of years, this trend has broken. You pay double or even quadruple the price for a relatively tenuous increase in performance.

We said that in 1982, and 1987, and 1993, and 1995, and 2001, 2003, 2003.5

You get the point.

There's always local optimization that leads to improvements. Look at the Apple M1 chip rollout as a prime example of that. Big/Little processors, on die RAM, shared memory with the GPU and Neural Engine, power integration with the OS.

LOTS of things that led to a big leap forward.

Re: ChatGPT could cost over $700k per day to operate

#93

Post by tinideiznaimnou »

seattle_spring wrote: post #85
consp wrote: post #64

Earlier quoted context omitted.

Don't generalize, I've never used it and never seen any reason to use it.

My grandma has never used the internet and sees no reason to use it. It would benefit her immensely to use it.

[dead]

Re: ChatGPT could cost over $700k per day to operate

#94

Post by rickdeckard »

capableweb wrote: post #62
gabereiser wrote: post #40

Earlier quoted context omitted.

that's ridiculous, OpenAI is paying. Granted Microsoft invested heavily into OpenAI but those are two separate financial transactions. Sure you can rationalize in your head that IN-OUT=DIFF but that's not how books are kept.

Why is that ridiculous? Cloud services gives companies "coupons" and free usage for X hours for a bunch of other companies, why wouldn't they do that for a company they invested heavily in?

Because that's not how it works. Even company cars of General Motors employees have to be purchased from General Motors.

Such "free usage" coupons are marketing activities to gain new customers, Microsoft already completed the "dating phase" with OpenAI. They surely don't pay list-price for Azure but it's surely also not free.

Moreover, as per Microsoft themselves, the 1bn USD investment into OpenAI carried the condition that Azure becomes the exclusive provider for cloud-services: https://blogs.microsoft.com/blog/2023/01/23/microsoftandopen...

It's not exclusive because it's free, it's exclusive because "we paid you 1bn USD to buy it from us"

Re: ChatGPT could cost over $700k per day to operate

#95

Post by bob1029 »

This sounds like a lot but I feel like optimizations are going to chop a zero off of that figure pretty quickly.

For example, why not cache user prompt/response pairs and use cosine distance for key lookup? You could probably find a way to do this at the edge without a whole lot of suffering. I suspect many of the prompts that hit the public API are effectively the same thing over and over every day. Why let that kind of traffic touch the expensive part of the architecture?

Re: ChatGPT could cost over $700k per day to operate

#96

Post by jermaustin1 »

octopoc wrote: post #71
jermaustin1 wrote: post #30

Earlier quoted context omitted.

I have started incorporating ChatGPT into my side project. I'm not integrating it into software or anything like that, but using it as a tool for product ideas, product photo ideas, funny social media captions, figuring out the best hashtags to use, product tagging and categorization. It has proven to be a great way to get everything to around 70%, then send off to my assistant for the remaining 30% of polish. So at…

I've been using it for product ideas as well. Occasionally it's brilliant but sometimes it's so bad it's funny, like when it suggested Seven-Year Itch for a perfume brand.

I actually like to make the bad ones. It is fun. It is an exercise in creative engineering. Especially since my side project is making home goods for stoners.

Re: ChatGPT could cost over $700k per day to operate

#97

Post by dmix »

capableweb wrote: post #20
zamnos wrote: post #16

Earlier quoted context omitted.

They're interesting numbers, but the linked article's cite amounts to: > ChatGPT could cost OpenAI up to $700,000 a day to run due to "expensive servers," an analyst told The Information. which, pardon me, but no shit. Before I break out my back of the envelope calculator, on how many biggest GPU instances in Azure that is, the real question is what their underlying assumptions are, and where they're getting them fro…

Considering that Microsoft is a huge investor in OpenAI, I'd be surprised they pay anything at all in reality.

At best OpenAI has negotiated a near 0 profit margin for Microsoft when paying for the services. But even that is unlikely given how much money/resources are involved. There's no scenario where it's free at that scale.

Re: ChatGPT could cost over $700k per day to operate

#98

Post by imranq »

warent wrote: post #49
nathancahill wrote: post #41

Earlier quoted context omitted.

Yeah think of all the labor hours, how many people do they have back there typing up the answers?

And who pays the tiny people in my magic light box to draw the answers on the window?!

I thought they were on contract? hopefully they don't go on strike again at the same time the power went out like last time

Re: ChatGPT could cost over $700k per day to operate

#99

Post by jyu »

Sorry for the tangent, wondering how you've used GPT to make your life better?

I'll start:

- making little scripts in shell / js / python that I'm not as fluent in. 5 min vs 1-3 hours

- explaining repos and apis instead of reading all the docs - help with debugging

- flushing out angles for new concepts that I did not previously consider (ex: how do you make a good decentralized exchange)

Re: ChatGPT could cost over $700k per day to operate

#100

Post by dmix »

bob1029 wrote: post #95

This sounds like a lot but I feel like optimizations are going to chop a zero off of that figure pretty quickly. For example, why not cache user prompt/response pairs and use cosine distance for key lookup? You could probably find a way to do this at the edge without a whole lot of suffering. I suspect many of the prompts that hit the public API are effectively the same thing over and over every day. Why let that kin…

It's not like a database you can cache though, the responses are non-deterministic, the responses you get can be different the next time you query it with the exact same prompt. That's part of the point of it being generative AI (vs a question/answer system that people imagine it is).
Post reply on HN