Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT could cost over $700k per day to operate

businessinsider.com

Post reply on HN
Print view View original

171–180 of 223 posts

Re: ChatGPT could cost over $700k per day to operate

#171

Post by javajosh »

Swizec wrote: post #70
javajosh wrote: post #52

Earlier quoted context omitted.

It's almost like ChatGPT is nerd-sniping us all. But we like it and it's useful to us.

Nerd-snipe and useful? That’s the holy grail! We’re having fun and we’re solving real problems that we have. Amazing. I’ve already incorporated it into my publishing process. My home-grown “cms” uses ChatGPT (via API) to write my article description, draft a twitter thread, and craft a “viral insight”. The latter is mostly useful to make sure my article even makes a point. Hoping to use it for a related articles feat…

> Nerd-snipe and useful? That’s the holy grail!

Yes. i think its a good software product! Maybe one of the best there ever was - I'm not surprised Sam Altman jumped the YC ship for this one, or that MS is particularly interested it. It gets ignored in all the other bru-haha but I'm excited to see what the really talented software teams of the world will be able to accomplish with an AI coding assisstent - and I don't mean just in the world of AI.

The activation energy for new software lowered by about 30% overnight, which is outrageously cool, and of course disquieting.

Re: ChatGPT could cost over $700k per day to operate

#172

Post by kolinko »

Ekaros wrote: post #84

Now I wonder what are the emissions. Is this technology net positive in long run for things like climate change?

This tech can more or less improve productivity in every sector of the economy, and that includes energy/climate change.

Personally, I've used it with success with R&D and due dil of projects related to climate change. LLMs (and progress in general) can help tremendously with switching to a sustainable economy.

Re: ChatGPT could cost over $700k per day to operate

#173

Post by kolinko »

roflyear wrote: post #150
pdksam wrote: post #13

Earlier quoted context omitted.

If anything performance should get better with time

What I mean is resources will be limited or models that are slightly worse will be released that will be much more cost effective but not quite as good. This is often the case with these types of technologies.

So far, with technologies, it's been that new tech is both cheaper and better than the previous one.

To not look far - gpt3.5 turbo.

Re: ChatGPT could cost over $700k per day to operate

#174

Post by kolinko »

marginalia_nu wrote: post #59
reitanqild wrote: post #55

Earlier quoted context omitted.

> 10% here and there is very small compared to the literal orders magnitude improvements during the reign of Moore's Law. I can't confirm it, but I noticed this comment says "gpu tech has beat Moore’s law for DNNs the last several years": https://news.ycombinator.com/item?id=35653231

We're actually at an inflection point where this isn't the case anymore. For a long time, GPU hardware basically became more powerful with each generation, but prices stayed roughly the same plus minus inflation. Last couple of years, this trend has broken. You pay double or even quadruple the price for a relatively tenuous increase in performance.

That's not true. You can buy Raspberry PI, which is 10x cheaper and 10x more powerful than the computers at the beginning of 2000s.

Ditto with mobile phones. iPhone may be more expensive than when it launched, but you can buy dirt-cheap chinese smartphones that have similar performance - if not higher to the first iPhones.

Re: ChatGPT could cost over $700k per day to operate

#175

Post by kolinko »

CuriouslyC wrote: post #102
marginalia_nu wrote: post #54

Earlier quoted context omitted.

This would be optimizations, which is not really the same thing as moore's law-like growth which was absolutely mind-boggling, like it's hard to even wrap your head around how fast tech was moving in that period since humans don't really grok exponentials too well, we just think they look like second degree polynomials.

Probabilistic computing offers the potential of a return to that pace of progress. We spend a lot of silicon on squashing things to 0/1 with error correction, but using analog voltages to carry information and relying on parameter redundancy for error correction could lead to much greater efficiency both in terms of OPS/mm^2 and OPS/watt.

I am wondering about this as well - wondering how difficult it would be to build an analog circuit for a small LLM (7B?). And wondering if anyone's working on that yet. Seems like an obvious avenue to huge efficiency gains.

Re: ChatGPT could cost over $700k per day to operate

#176

Post by iloveCS »

moffkalast wrote: post #128
cft wrote: post #104

Earlier quoted context omitted.

Bing gives you much shorter answers, often without code examples

I'm sorry but I have to end this conversation, I have been a good Bing and you have been a bad user. Thank you for your cooperation. Goodbye.

[dead]

Re: ChatGPT could cost over $700k per day to operate

#177

Post by dqft »

bemmu wrote: post #108
kylecazar wrote: post #90

Earlier quoted context omitted.

There are so many subscription products these days that it makes sense to assess the value of each carefully before committing. If you're like me you tend to forget you're paying for something, and have to do a yearly purge. It's very easy to slowly but surely rack up hundreds of dollars a month in subscription services.

My habit is to cancel right away when I start a subscription. You can still use the service for the month you paid for, and it's easy to restart it if you do still find yourself attempting to use it a month later.

Substack disappointed me because you can't do this. They remove content access if you do not have an "active" subscription. You don't have to pay again to "reactivate", but the paid content is locked until you do. This is obviously intended dark behavior because when I went to fully unsub later in the month they gave me a special dialog and a free month hoping for me to forget again this month. Very lame.

Re: ChatGPT could cost over $700k per day to operate

#179

Post by bob1029 »

dmix wrote: post #100
bob1029 wrote: post #95

This sounds like a lot but I feel like optimizations are going to chop a zero off of that figure pretty quickly. For example, why not cache user prompt/response pairs and use cosine distance for key lookup? You could probably find a way to do this at the edge without a whole lot of suffering. I suspect many of the prompts that hit the public API are effectively the same thing over and over every day. Why let that kin…

It's not like a database you can cache though, the responses are non-deterministic, the responses you get can be different the next time you query it with the exact same prompt. That's part of the point of it being generative AI (vs a question/answer system that people imagine it is).

> That's part of the point of it being generative AI (vs a question/answer system that people imagine it is).

The point I am trying to make is that not all use cases for ChatGPT are generative. There are a lot of Q&A use cases today despite the fact that these are so far beneath its true capabilities. These items could be dealt with using more economical means.

"give me a recipe for XYZ" should not require a GPU for the first turn response, much like typing in an offensive manifesto returns a boilerplate "as an AI language model..." response.

Granted, if the user then types something like "please translate the recipe to Spanish and increase the amounts by 33%", we would have to reach for the generative model. But, how many real-world users are satisfied with some simple 1-turn response and go about their day?

Re: ChatGPT could cost over $700k per day to operate

#180

Post by williamcotton »

octopoc wrote: post #71
jermaustin1 wrote: post #30

Earlier quoted context omitted.

I have started incorporating ChatGPT into my side project. I'm not integrating it into software or anything like that, but using it as a tool for product ideas, product photo ideas, funny social media captions, figuring out the best hashtags to use, product tagging and categorization. It has proven to be a great way to get everything to around 70%, then send off to my assistant for the remaining 30% of polish. So at…

I've been using it for product ideas as well. Occasionally it's brilliant but sometimes it's so bad it's funny, like when it suggested Seven-Year Itch for a perfume brand.

That is a great name for a perfume brand! It’s also the name of a comedy by Billy Wilder staring Marylyn Monroe. Market it to rich old ladies.
Post reply on HN