Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is Chat GPT so expensive to operate?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 54 posts

Re: Why is Chat GPT so expensive to operate?

#21

Post by wordpad25 »

smnrchrds wrote: post #19
mr_00ff00 wrote: post #14

Earlier quoted context omitted.

Really makes you appreciate the brain, which presumably operates with some sort of similar demand.

Another mind-boggling thing about brain is how little power it uses to do all the complex things it does.

calories are a unit of energy, so it’s a straight forward comparison

if we assume that a computer can be powered by 100 watts, over a day it will use 2.4 kW h, which is about 2000 Calories

GPU will consume a lot more, but we aren’t that far off in efficiency

Re: Why is Chat GPT so expensive to operate?

#22

Post by pattrn »

wordpad25 wrote: post #21
smnrchrds wrote: post #19

Earlier quoted context omitted.

Another mind-boggling thing about brain is how little power it uses to do all the complex things it does.

calories are a unit of energy, so it’s a straight forward comparison if we assume that a computer can be powered by 100 watts, over a day it will use 2.4 kW h, which is about 2000 Calories GPU will consume a lot more, but we aren’t that far off in efficiency

Doesn't that assume 100% of a human's daily calories burn is due to brain activity?

Re: Why is Chat GPT so expensive to operate?

#23

Post by unsupp0rted »

pattrn wrote: post #22
wordpad25 wrote: post #21

Earlier quoted context omitted.

calories are a unit of energy, so it’s a straight forward comparison if we assume that a computer can be powered by 100 watts, over a day it will use 2.4 kW h, which is about 2000 Calories GPU will consume a lot more, but we aren’t that far off in efficiency

Doesn't that assume 100% of a human's daily calories burn is due to brain activity?

The brain uses about 20% of a human's calories. It's not 100%, but it's a substantial fraction.

Re: Why is Chat GPT so expensive to operate?

#24

Post by cypress66 »

sidibe wrote: post #17
thealch3m1st wrote: post #16

Could models like chatGPT run on hardware like the Tesla Dojo ? If so maybe Elon should donate some...

Does dojo even exist? He kept talking about how it was almost ready a couple years ago, no word since which is strange from such a braggart

They unveiled it on AI day 2021, talked more about it on AI day 2022, and in theory should start operating Q1 2023.

Re: Why is Chat GPT so expensive to operate?

#25

Post by faebi »

scarface74 wrote: post #15

My question is how long will it be before the average high end computer can run it? How long before your average smart phone? Memory shipped with computers have been stagnate for a decade

Maybe that will be the next use case to make larger amounts of memory mainstream. At the same time, somehow Tesla still manages to cram more and more neural nets into that small memory. So it could also be that many neural nets are just not really efficient yet.

Re: Why is Chat GPT so expensive to operate?

#26

Post by _8j50 »

mr_00ff00 wrote: post #14
vineyardmike wrote: post #11

All these answers are good, but I can share more concrete numbers… Meta released their OPT model which they claim is comparable to the GPT-3 model. Guidance for running that model [1] suggests a LOT of memory - at least 350GB of gpu memory which is roughly 4 A1000s, which are pricy. Running this on AWS with the above suggestion would cost $25/hr - just for one model running. That’s almost $0.50 a minute. If you imagi…

Really makes you appreciate the brain, which presumably operates with some sort of similar demand.

To have ML produce meaningful content you need tp give it some input or a sense of what the outcome should be and this is after billions of trial and errors.

Yet people these days believe something like the brain was bruteforced by nature into an accidental existence.

Re: Why is Chat GPT so expensive to operate?

#27

Post by melagonster »

_8j50 wrote: post #26
mr_00ff00 wrote: post #14

Earlier quoted context omitted.

Really makes you appreciate the brain, which presumably operates with some sort of similar demand.

To have ML produce meaningful content you need tp give it some input or a sense of what the outcome should be and this is after billions of trial and errors. Yet people these days believe something like the brain was bruteforced by nature into an accidental existence.

it is natural selection. this is most famous mechanism of evolution.

Re: Why is Chat GPT so expensive to operate?

#28

Post by est31 »

scarface74 wrote: post #15

My question is how long will it be before the average high end computer can run it? How long before your average smart phone? Memory shipped with computers have been stagnate for a decade

People are already trying to put cutting edge models onto consumer hardware: https://news.ycombinator.com/item?id=32678664

We live in a really exciting age :). Local AI models will also finally give Microsoft reasons again to require hardware for coming Windows versions. Now they have to require obscure security chips and stuff but in the future they might have some local cortana thingy or something that requires a certain amount of computational power.

Post reply on HN