Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Researchers run high-performing LLM on the energy needed to power a lightbulb

news.ucsc.edu

Post reply on HN
Print view View original

11–20 of 72 posts

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#11

Post by geewee »

croes wrote: post #6
blixt wrote: post #2

> It costs $700,000 per day in energy costs to run ChatGPT 3.5, according to recent estimates, and leaves behind a massive carbon footprint in the process. Compared to what? I wouldn't defend LLMs as "worth their electricity" quite yet, and they are definitely less efficient than a lot of other software, but I'd still like to see how this compares to gaming consoles, or email servers, the advertising industry hosting…

>Just doesn't seem worth pointing out the carbon footprint of AI just yet. Of course it does. It's not like AI replaced anything you mentioned. Its carbon footprint comes on top of it. The benefit is secondary if the end result just means more carbon dioxide.

> It's not like AI replaced anything you mentioned.

I mean it's true it hasn't replaced anything the OP mentioned, but it has definitely replaced parts of the compute that I would normally use for e.g. searching.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#12

Post by blixt »

croes wrote: post #6
blixt wrote: post #2

> It costs $700,000 per day in energy costs to run ChatGPT 3.5, according to recent estimates, and leaves behind a massive carbon footprint in the process. Compared to what? I wouldn't defend LLMs as "worth their electricity" quite yet, and they are definitely less efficient than a lot of other software, but I'd still like to see how this compares to gaming consoles, or email servers, the advertising industry hosting…

>Just doesn't seem worth pointing out the carbon footprint of AI just yet. Of course it does. It's not like AI replaced anything you mentioned. Its carbon footprint comes on top of it. The benefit is secondary if the end result just means more carbon dioxide.

I meant more that the % is so low that even if the usage is on top of all other usage (not a completely clear statement to make), it's like starting to mention any other thing in the long tail of technology leaving behind a "massive carbon footprint". Yes, it matters, especially if you were making a report focused on sources of carbon footprint, but in general, saying "AI carbon footprint is bad" just seems like wanting to give it a bad name. In reality AI tech doesn't seem to be such a big contributor percentage-wise. Of course it should still be optimized, not arguing that.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#13

Post by Aurornis »

The press release is devoid of useful information, unsurprisingly. You can run an LLM under almost any energy envelope if you’re willing to wait long enough for the result. Total energy consumed and the time difference are the more important metrics.

The actual paper is here: https://arxiv.org/abs/2406.02528

The key part from the summary:

> To properly quantify the efficiency of our architecture, we build a custom hardware solution on an FPGA which exploits lightweight operations beyond what GPUs are capable of. We processed billion-parameter scale models at 13W beyond human readable throughput, moving LLMs closer to brain-like efficiency.

There is a lot of unnecessary obfuscation of the numbers going on in the abstract as well, which is unfortunate. Instead of quoting the numbers they call it “billion-parameter scale” and “beyond human readable throughout”.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#14

Post by FpUser »

Aurornis wrote: post #13

The press release is devoid of useful information, unsurprisingly. You can run an LLM under almost any energy envelope if you’re willing to wait long enough for the result. Total energy consumed and the time difference are the more important metrics. The actual paper is here: https://arxiv.org/abs/2406.02528 The key part from the summary: > To properly quantify the efficiency of our architecture, we build a custom ha…

They did say that the answer is being produced faster than the human can read.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#15

Post by jimbobthrowawy »

Retr0id wrote: post #4

That's 13 Watts apparently, in non-American units.

Is there an American unit for power? I thought J/s is universal. 0.017433 Horsepower?

I had expected them to make the title more clickbaity, but that number is about right for a modern lightbulb.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#16

Post by Arainach »

Aurornis wrote: post #9
blixt wrote: post #2

> It costs $700,000 per day in energy costs to run ChatGPT 3.5, according to recent estimates, and leaves behind a massive carbon footprint in the process. Compared to what? I wouldn't defend LLMs as "worth their electricity" quite yet, and they are definitely less efficient than a lot of other software, but I'd still like to see how this compares to gaming consoles, or email servers, the advertising industry hosting…

ChatGPT is very popular, with many users. Any article citing the power usage without calculating it in terms of users of queries is just trying to push an agenda by omitting how many people are using it.

When you make something more available/cheaper, overall usage often goes up.

Overall energy use is an important metric regardless of energy per task.

Airplanes are WAY more efficient per passenger than they were in the past, but it's still valid to express concern over the energy usage and pollutants of air travel with so many more routes being flown.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#17

Post by Aurornis »

cs702 wrote: post #10

Paper: https://arxiv.org/abs/2406.02528 -- always better than a press release. Code: https://github.com/ridgerchu/matmulfreellm --- Like others before them, the authors train LLMs using parameters consisting of ternary digits, or trits, with values in {-1, 0, 1}. What's new is that the authors then build a custom hardware solution on an FPGA and run billion-parameter LLMs consuming only 13W, moving LLM inference clos…

The claim about moving closer to brain-like efficiency conveniently omits how that model compares to modern LLMs. You can put together a toy LLM that is much smaller and more efficient than ChatGPT but isn’t as useful and call it “more efficient”, but that’s not useful in practice.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#18

Post by jstanley »

blixt wrote: post #2

> It costs $700,000 per day in energy costs to run ChatGPT 3.5, according to recent estimates, and leaves behind a massive carbon footprint in the process. Compared to what? I wouldn't defend LLMs as "worth their electricity" quite yet, and they are definitely less efficient than a lot of other software, but I'd still like to see how this compares to gaming consoles, or email servers, the advertising industry hosting…

> they are definitely less efficient than a lot of other software

This is not a fair claim to make. Is a milling machine less efficient than a clock?

It does a different thing, it's not really comparable.

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#19

Post by eli »

Aurornis wrote: post #9
blixt wrote: post #2

> It costs $700,000 per day in energy costs to run ChatGPT 3.5, according to recent estimates, and leaves behind a massive carbon footprint in the process. Compared to what? I wouldn't defend LLMs as "worth their electricity" quite yet, and they are definitely less efficient than a lot of other software, but I'd still like to see how this compares to gaming consoles, or email servers, the advertising industry hosting…

ChatGPT is very popular, with many users. Any article citing the power usage without calculating it in terms of users of queries is just trying to push an agenda by omitting how many people are using it.

How many people are using it because it's being offered at far below operating costs, even before you factor in the externalities of massive energy use?

Re: Researchers run high-performing LLM on the energy needed to power a lightbulb

#20

Post by refulgentis »

Aurornis wrote: post #9
blixt wrote: post #2

> It costs $700,000 per day in energy costs to run ChatGPT 3.5, according to recent estimates, and leaves behind a massive carbon footprint in the process. Compared to what? I wouldn't defend LLMs as "worth their electricity" quite yet, and they are definitely less efficient than a lot of other software, but I'd still like to see how this compares to gaming consoles, or email servers, the advertising industry hosting…

ChatGPT is very popular, with many users. Any article citing the power usage without calculating it in terms of users of queries is just trying to push an agenda by omitting how many people are using it.

You had me till your thesis.

"Push an agenda"?

If they inserted a couple paragraphs saying "we estimate about 200M users a day, etc. etc.", would that add or detract from the article?

It's not relevant how they derived the number when you're reading, you only need an order of magnitude estimate, rest is distraction.

Post reply on HN