Researchers run high-performing LLM on the energy needed to power a lightbulb
1–10 of 72 posts
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#2> It costs $700,000 per day in energy costs to run ChatGPT 3.5, according to recent estimates, and leaves behind a massive carbon footprint in the process.
Compared to what? I wouldn't defend LLMs as "worth their electricity" quite yet, and they are definitely less efficient than a lot of other software, but I'd still like to see how this compares to gaming consoles, or email servers, the advertising industry hosting costs, cryptocurrency, and so on. Just doesn't seem worth pointing out the carbon footprint of AI just yet.
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#3It looks like this is a quantization method to flatten matrices for vector addition. Can anyone explain how this could allow LLMs to reach current benchmarks without losing performance?
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#4That's 13 Watts apparently, in non-American units.
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#5> It costs $700,000 per day in energy costs to run ChatGPT 3.5, according to recent estimates, and leaves behind a massive carbon footprint in the process. Compared to what? I wouldn't defend LLMs as "worth their electricity" quite yet, and they are definitely less efficient than a lot of other software, but I'd still like to see how this compares to gaming consoles, or email servers, the advertising industry hosting…
The only source I can find for this estimate is from a year ago. I feel like efficiency has gone up by a lot since then
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#6> It costs $700,000 per day in energy costs to run ChatGPT 3.5, according to recent estimates, and leaves behind a massive carbon footprint in the process. Compared to what? I wouldn't defend LLMs as "worth their electricity" quite yet, and they are definitely less efficient than a lot of other software, but I'd still like to see how this compares to gaming consoles, or email servers, the advertising industry hosting…
>Just doesn't seem worth pointing out the carbon footprint of AI just yet.
Of course it does. It's not like AI replaced anything you mentioned. Its carbon footprint comes on top of it.
The benefit is secondary if the end result just means more carbon dioxide.
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#7> It costs $700,000 per day in energy costs to run ChatGPT 3.5, according to recent estimates, and leaves behind a massive carbon footprint in the process. Compared to what? I wouldn't defend LLMs as "worth their electricity" quite yet, and they are definitely less efficient than a lot of other software, but I'd still like to see how this compares to gaming consoles, or email servers, the advertising industry hosting…
The only source I can find for this estimate is from a year ago. I feel like efficiency has gone up by a lot since then
Same as usage
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#8It looks like this is a quantization method to flatten matrices for vector addition. Can anyone explain how this could allow LLMs to reach current benchmarks without losing performance?
The article mentions two things. Quantization (which everyone is already doing). There's bitnet, or you can Google 1 bit quantization (it's not really 1 bit, it's ternary as the article says).
Once you do that, a dot product is just addition/subtraction. Matrix multiplications are just dot products, so you've removed multiplication.
Then they built custom hardware that presumably only does that operation and doesn't use much electricity
Since this is just "more aggressive quantization" it's not too surprising that it reaches similar performance to other quantized models.
The network should structurally look the same as any other llm, it's still using transformers etc (afaiu)
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#9> It costs $700,000 per day in energy costs to run ChatGPT 3.5, according to recent estimates, and leaves behind a massive carbon footprint in the process. Compared to what? I wouldn't defend LLMs as "worth their electricity" quite yet, and they are definitely less efficient than a lot of other software, but I'd still like to see how this compares to gaming consoles, or email servers, the advertising industry hosting…
ChatGPT is very popular, with many users.
Any article citing the power usage without calculating it in terms of users of queries is just trying to push an agenda by omitting how many people are using it.
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#10Paper: https://arxiv.org/abs/2406.02528 -- always better than a press release.
Code: https://github.com/ridgerchu/matmulfreellm
---
Like others before them, the authors train LLMs using parameters consisting of ternary digits, or trits, with values in {-1, 0, 1}.
What's new is that the authors then build a custom hardware solution on an FPGA and run billion-parameter LLMs consuming only 13W, moving LLM inference closer to brain-like efficiency.
Sure, it's on an FPGA, and it's only a lab experiment, but we're talking about an early proof of concept, not a commercial product.
As far as I know, this is the first energy-efficient hardware implementation of tritwise LLMs. That seems like a pretty big deal to me.