Incandescent I presume. Mistral-7b on a Nvidia 3060 draws about 100-odd watts of power.
Didn't read the article I presume.
Researchers run high-performing LLM on the energy needed to power a lightbulb
41–50 of 72 posts
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#42Earlier quoted context omitted.
The only source I can find for this estimate is from a year ago. I feel like efficiency has gone up by a lot since then
Same as usage
Yep. See Jevons paradox
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#43The press release is devoid of useful information, unsurprisingly. You can run an LLM under almost any energy envelope if you’re willing to wait long enough for the result. Total energy consumed and the time difference are the more important metrics. The actual paper is here: https://arxiv.org/abs/2406.02528 The key part from the summary: > To properly quantify the efficiency of our architecture, we build a custom ha…
Yeah... They are using a single-core 13W measurement to project out. For a 64x parallelization - no mention of any overhead due to parallelization or power needs of the supporting hardware. This is a key quote for me (page 12 of the PDF):
> The 1.3B parameter model, where L = 24 and d = 2048, has a projected runtime of 42ms, and a throughput of 23.8 tokens per second.
e.g. 64 x 13.67W = 874 Watts to run a 1.3B model at 23.8 t/s... I'm pretty sure my phone can do way better than that! Even half that power given their assertions in the table are still overpowered for such a small model.
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#44The press release is devoid of useful information, unsurprisingly. You can run an LLM under almost any energy envelope if you’re willing to wait long enough for the result. Total energy consumed and the time difference are the more important metrics. The actual paper is here: https://arxiv.org/abs/2406.02528 The key part from the summary: > To properly quantify the efficiency of our architecture, we build a custom ha…
They did say that the answer is being produced faster than the human can read.
yes, human average reading speed is ~250 words per minute, but to be fair, it's not a widely known stat.
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#45Paper: https://arxiv.org/abs/2406.02528 -- always better than a press release. Code: https://github.com/ridgerchu/matmulfreellm --- Like others before them, the authors train LLMs using parameters consisting of ternary digits, or trits, with values in {-1, 0, 1}. What's new is that the authors then build a custom hardware solution on an FPGA and run billion-parameter LLMs consuming only 13W, moving LLM inference clos…
The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs. You can put together a toy LLM that is much smaller and more efficient than ChatGPT but isn’t as useful and call it “more efficient”, but that’s not useful in practice.
I’ve heard some claims that to get closer to brain-like energy efficiency you’d need to use a spiking neural network
https://en.wikipedia.org/wiki/Spiking_neural_network
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#46And for candle power?
Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#47Re: Researchers run high-performing LLM on the energy needed to power a lightbulb
#48It’s cool that my ANN course from twenty years back gives me enough background to understand this and quantization.
My professor at the time was at his last leg in impact, when ANNs were looked down on right before someone had the bright idea of using video cards.
I hope he’s doing well/retired on a high note.