Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Researchers run high-performing LLM on the energy needed to power a lightbulb

news.ucsc.edu

Post reply on HN
Print view View original

61–70 of 72 posts

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#61

Post by dustypotato »

Escapado wrote: post #37

> "For the largest model size of 13B parameters, the MatMul-free LM uses only 4.19 GB of GPU memory and has a latency of 695.48 ms, whereas Transformer++ requires 48.50 GB of memory and exhibits a latency of 3183.10 ms" That's a really, _really_ big difference in memory usage and since this scales sub-linear (300M param model uses 0.21GB, 13B model uses 4.19B) a 70B model would fit on an RTX 4090. I think currently p…

> a 70B model would fit on an RTX 4090 With this technique, it will not, because it uses a custom FPGA, not consumer GPUs

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#62

Post by larodi »

cs702 wrote: post #10

Paper: https://arxiv.org/abs/2406.02528 -- always better than a press release. Code: https://github.com/ridgerchu/matmulfreellm --- Like others before them, the authors train LLMs using parameters consisting of ternary digits, or trits, with values in {-1, 0, 1}. What's new is that the authors then build a custom hardware solution on an FPGA and run billion-parameter LLMs consuming only 13W, moving LLM inference clos…

to me the most interesting part in this all is the quantization used. hardware doing LLMs is likely to be the new norm in few years anyway. with retrofitting existing hardware to use say USBs running LLM accelerators and alike...

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#63

Post by cs702 »

avisser wrote: post #55
cs702 wrote: post #24

Earlier quoted context omitted.

> The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs. I disagree. The authors aren't conveniently omitting anything. They show all details in a comparison against LLama models. Moreover, all evidence I've seen so far suggests that tritwise models can scale up to state-of-the-art sizes. --- PS. I'm talking about the paper, not the fluffy press release.

I took the critique as being against OP, not the paper.

Ah, that makes more sense :-)

Thanks for pointing it out!

PS. I added a PS to my comment above.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#64

Post by dwallin »

dustypotato wrote: post #61
Escapado wrote: post #37

> "For the largest model size of 13B parameters, the MatMul-free LM uses only 4.19 GB of GPU memory and has a latency of 695.48 ms, whereas Transformer++ requires 48.50 GB of memory and exhibits a latency of 3183.10 ms" That's a really, _really_ big difference in memory usage and since this scales sub-linear (300M param model uses 0.21GB, 13B model uses 4.19B) a 70B model would fit on an RTX 4090. I think currently p…

> a 70B model would fit on an RTX 4090 With this technique, it will not, because it uses a custom FPGA, not consumer GPUs

They tested their models on both a conventional gpu as well as a custom FPGA.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#65

Post by fuzzfactor »

lawlessone wrote: post #59
cs702 wrote: post #10

Paper: https://arxiv.org/abs/2406.02528 -- always better than a press release. Code: https://github.com/ridgerchu/matmulfreellm --- Like others before them, the authors train LLMs using parameters consisting of ternary digits, or trits, with values in {-1, 0, 1}. What's new is that the authors then build a custom hardware solution on an FPGA and run billion-parameter LLMs consuming only 13W, moving LLM inference clos…

> moving LLM inference closer to brain-like efficiency. Yeah but the brain does more than predictive text.

At least they're aware of what it is to be on the wrong track.

According to the researchers,

>all we had to do was fundamentally change how neural networks work,

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#66

Post by cpeterso »

cs702 wrote: post #10

Paper: https://arxiv.org/abs/2406.02528 -- always better than a press release. Code: https://github.com/ridgerchu/matmulfreellm --- Like others before them, the authors train LLMs using parameters consisting of ternary digits, or trits, with values in {-1, 0, 1}. What's new is that the authors then build a custom hardware solution on an FPGA and run billion-parameter LLMs consuming only 13W, moving LLM inference clos…

I assume the ternary weight’s memory representation requires two bits, so why do they use only three values instead of four? OTOH, I’m not sure what fourth value would useful for LLM models doing math with -1, 0, and 1. Infinity? NaN?

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#69

Post by ag_guy »

robertlagrant wrote: post #40
croes wrote: post #25

Earlier quoted context omitted.

But do you now spend less time on the computer?

If people are activating Google's servers less, that's some energy saved. I don't know how it compares, but I guess OpenAI aren't running a live bidding war with advertisers on every request.

> If people are activating Google's servers less, that's some energy saved

Instead of google's server people are accessing OpenAi servers, which wastes even more power. How is that any better?

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#70

Post by shivaluminaire »

pbnjay wrote: post #43
Aurornis wrote: post #13

The press release is devoid of useful information, unsurprisingly. You can run an LLM under almost any energy envelope if you’re willing to wait long enough for the result. Total energy consumed and the time difference are the more important metrics. The actual paper is here: https://arxiv.org/abs/2406.02528 The key part from the summary: > To properly quantify the efficiency of our architecture, we build a custom ha…

Yeah... They are using a single-core 13W measurement to project out. For a 64x parallelization - no mention of any overhead due to parallelization or power needs of the supporting hardware. This is a key quote for me (page 12 of the PDF): > The 1.3B parameter model, where L = 24 and d = 2048, has a projected runtime of 42ms, and a throughput of 23.8 tokens per second. e.g. 64 x 13.67W = 874 Watts to run a 1.3B model…

When you multiply by 64 you also get 64 times more tokens per second!! Your math is wrong.
Post reply on HN