Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Researchers run high-performing LLM on the energy needed to power a lightbulb

news.ucsc.edu

Post reply on HN
Print view View original

51–60 of 72 posts

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#51

Post by ameesdotme »

XorNot wrote: post #39
blixt wrote: post #12

Earlier quoted context omitted.

I meant more that the % is so low that even if the usage is on top of all other usage (not a completely clear statement to make), it's like starting to mention any other thing in the long tail of technology leaving behind a "massive carbon footprint". Yes, it matters, especially if you were making a report focused on sources of carbon footprint, but in general, saying "AI carbon footprint is bad" just seems like want…

"carbon footprint" for any computational technology is a ridiculous notion anyway: it's all electricity, and electricity is source independent.

Just because the carrier of energy is source independent, doesn't mean the consumer of that energy is not responsible for the carbon emissions of its production. Since we're talking hundreds of TWh[1], the policies of those consumers can have a massive impact on global emissions.

[1] https://www.iea.org/energy-system/buildings/data-centres-and...

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#52

Post by mensetmanusman »

boto3 wrote: post #44
FpUser wrote: post #14

Earlier quoted context omitted.

They did say that the answer is being produced faster than the human can read.

yes, human average reading speed is ~250 words per minute, but to be fair, it's not a widely known stat.

Speaking is 150 wpm

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#53

Post by moffkalast »

Lightbulbs are probably not the best thing to compare against, they're famously high in energy consumption and terribly inefficient. Even LEDs are absolutely awful and barely crack 30% total efficiency, most of what they make is heat.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#54

Post by onlyrealcuzzo »

boto3 wrote: post #44
FpUser wrote: post #14

Earlier quoted context omitted.

They did say that the answer is being produced faster than the human can read.

yes, human average reading speed is ~250 words per minute, but to be fair, it's not a widely known stat.

It's almost as if our brain is optimized for taking in more signals than just words on a page.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#55

Post by avisser »

cs702 wrote: post #24
Aurornis wrote: post #17

Earlier quoted context omitted.

The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs. You can put together a toy LLM that is much smaller and more efficient than ChatGPT but isn’t as useful and call it “more efficient”, but that’s not useful in practice.

> The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs. I disagree. The authors aren't conveniently omitting anything. They show all details in a comparison against LLama models. Moreover, all evidence I've seen so far suggests that tritwise models can scale up to state-of-the-art sizes. --- PS. I'm talking about the paper, not the fluffy press release.

I took the critique as being against OP, not the paper.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#59

Post by lawlessone »

cs702 wrote: post #10

Paper: https://arxiv.org/abs/2406.02528 -- always better than a press release. Code: https://github.com/ridgerchu/matmulfreellm --- Like others before them, the authors train LLMs using parameters consisting of ternary digits, or trits, with values in {-1, 0, 1}. What's new is that the authors then build a custom hardware solution on an FPGA and run billion-parameter LLMs consuming only 13W, moving LLM inference clos…

> moving LLM inference closer to brain-like efficiency.

Yeah but the brain does more than predictive text.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#60

Post by airspresso »

bottom999mottob wrote: post #3

It looks like this is a quantization method to flatten matrices for vector addition. Can anyone explain how this could allow LLMs to reach current benchmarks without losing performance?

Very sceptical of the claims that they don't lose any performance. Sounds like wishful thinking without enough effort put into measuring the performance loss that has to be there due to the heavy quantization. They even dropped computing the entire matrix addition, only focusing on some parts of it. If the benchmarks used don't show a quality drop, then that's because those benchmarks are not able to properly measure said quality drop. (edit: typo)
Post reply on HN