Incandescent I presume. Mistral-7b on a Nvidia 3060 draws about 100-odd watts of power.
Researchers run high-performing LLM on the energy needed to power a lightbulb
31–40 of 72 posts
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#32Incandescent I presume. Mistral-7b on a Nvidia 3060 draws about 100-odd watts of power.
Why presume? It's in the article. It's 13 watts.
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#33For reference, this is a real prototype for Bitnet 1.58b which uses (-1,0,1) as the weights which simplifies the matrix multiplication [0].
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#34Earlier quoted context omitted.
I asked GPT-4 to estimate how much CO2 this likely emits, in units of "typical car usage in a city." It suggests this emits roughly as much CO2 as Reno or Des Moines. That's staggering, but there are about 100 cities this size in the US, so decreasing car usage 1% would more than offset this. I know this is a bizarre comparison to make, but CO2 emissions are fungible.
> I asked GPT-4 What makes you confident it gave you an accurate answer?
"I asked GPT" is the new "I saw it on TV."
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#35> It costs $700,000 per day in energy costs to run ChatGPT 3.5, according to recent estimates, and leaves behind a massive carbon footprint in the process. Compared to what? I wouldn't defend LLMs as "worth their electricity" quite yet, and they are definitely less efficient than a lot of other software, but I'd still like to see how this compares to gaming consoles, or email servers, the advertising industry hosting…
> they are definitely less efficient than a lot of other software This is not a fair claim to make. Is a milling machine less efficient than a clock? It does a different thing, it's not really comparable.
It’s fair in some cases, but indeed - not in others - e.g., code completion with LLMs is a time saver and worth paying for, compared to any other tech out there. At least this is how I see it, I know some people say it introduces mistakes but this is up for debate.
I do think if we look at translation tasks, grammar correction, information look up, etc. it adds a competitive convenience factor but I can’t say that running state of the art GPUs at very high wattages for up to a minute to do what specialized software can enable you to do by running for some milliseconds on much lower wattages isn’t less efficient. I’m referring to running multiple Google searches yourself to answer a question, or using a more traditional translation service, spell checker, and so on.
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#36I'm just curious how you close timing on a billion parameter model! I used to TA a digital design course that heavily involved FPGAs, and any kind of image or sprite usage, even on the order of megabytes, would crank up compile times insanely and sometimes even fail 30-45min into the build!
If anyone can offer insight that would be greatly appreciated
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#37> "For the largest model size of 13B parameters, the MatMul-free LM uses only 4.19 GB of GPU memory and has a latency of 695.48 ms, whereas Transformer++ requires 48.50 GB of memory and exhibits a latency of 3183.10 ms"
That's a really, _really_ big difference in memory usage and since this scales sub-linear (300M param model uses 0.21GB, 13B model uses 4.19B) a 70B model would fit on an RTX 4090. I think currently people often run 34B Models with 4bit quants on that so I would like to see some larger models trained on more tokens with this approach.
Also their 2.7B Model took 173hours on 8 NVIDIA H100 GPUs and that also seems to roughly scale linearly with the parameter size, so a company with access to a small cluster of those DGX pods (say 8) could train such a model in about 30 days - though the 100B token training set might be lackluster for SotA but maybe someone else could chime in on that.
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#38> It costs $700,000 per day in energy costs to run ChatGPT 3.5, according to recent estimates, and leaves behind a massive carbon footprint in the process. Compared to what? I wouldn't defend LLMs as "worth their electricity" quite yet, and they are definitely less efficient than a lot of other software, but I'd still like to see how this compares to gaming consoles, or email servers, the advertising industry hosting…
ChatGPT is very popular, with many users. Any article citing the power usage without calculating it in terms of users of queries is just trying to push an agenda by omitting how many people are using it.
Yeah, but the same 'number of users' claim could be fired at tiktok, facebook, linkedin, instagram, myriad other similarly dubious endeavours.
Active user count is not necessarily correlated to worthwhile consumption of resources.
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#39Earlier quoted context omitted.
>Just doesn't seem worth pointing out the carbon footprint of AI just yet. Of course it does. It's not like AI replaced anything you mentioned. Its carbon footprint comes on top of it. The benefit is secondary if the end result just means more carbon dioxide.
I meant more that the % is so low that even if the usage is on top of all other usage (not a completely clear statement to make), it's like starting to mention any other thing in the long tail of technology leaving behind a "massive carbon footprint". Yes, it matters, especially if you were making a report focused on sources of carbon footprint, but in general, saying "AI carbon footprint is bad" just seems like want…
"carbon footprint" for any computational technology is a ridiculous notion anyway: it's all electricity, and electricity is source independent.
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#40Earlier quoted context omitted.
> It's not like AI replaced anything you mentioned. I mean it's true it hasn't replaced anything the OP mentioned, but it has definitely replaced parts of the compute that I would normally use for e.g. searching.
But do you now spend less time on the computer?
If people are activating Google's servers less, that's some energy saved. I don't know how it compares, but I guess OpenAI aren't running a live bidding war with advertisers on every request.