Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI engineers claim new algorithm reduces AI power consumption by 95%

tomshardware.com

Post reply on HN
Print view View original

31–40 of 174 posts

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#31

Post by kayo_20211030 »

manquer wrote: post #22
kayo_20211030 wrote: post #10

Extraordinary claims require extraordinary evidence. Maybe it's possible, but consider that some really smart people, in many different groups, have been working diligently in this space for quite a while; so claims of 95% savings on energy costs _with equivalent performance_ is in the extraordinary category. Of course, we'll see when the tide goes out.

It is a click bait headline the claim itself is not extraordinary. the preprint from arxiv was posted here some time back . The 95% gains is specifically only for multiplication operations, inference is compute light and memory heavy in the first place so the actual gains would be far less smaller . Tech journalism (all journalism really) can hardly be trusted to publish grounded news with the focus on clicks and rev…

Thank you. That makes sense.

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#34

Post by fuzzfactor »

crazygringo wrote: post #20
onlyrealcuzzo wrote: post #17

Earlier quoted context omitted.

Does this mean you can train efficiently without GPUs? Presumably there will be a lot of interest.

No. But it does potentially mean that either current or future-tweaked GPUs could run a lot more efficiently -- meaning much faster or with much less energy consumption. You still need the GPU parallelism though.

I had a feeling it had to be something like massive waste due to a misguided feature of the algorithms that shouldn't have been there in the first place.

Once the "math is done" quite likely it would have paid off better than most investments for the top people to have spent a few short years working with grossly underpowered hardware until they could come up with amazing results there before scaling up. Rather than grossly overpowered hardware before there was even deep understanding of the underlying processes.

When you think about it, what we have seen from the latest ultra-high-powered "thinking" machines is truly so impressive. But if you are trying to fool somebody into believing that it's a real person it's still not "quite" there.

Maybe a good benchmark would be to take a regular PC, and without reliance on AI just pull out all the stops and put all the effort into fakery itself. No holds barred, any trick you can think of. See what the electronics is capable of this way. There are some smart engineers, this would only take a few years but looks like it would have been a lot more affordable.

Then with the same hardware if an AI alternative is not as convincing, something has got to be wrong.

It's good to find out this type of thing before you go overboard.

Regardless of speed or power, I never could have gotten an 8-bit computer to match the output of a 32-bit floating-point algorithm by using floating-point myself. Integers all the way and place the decimal where it's supposed to be when you're done.

Once it's really figured out, how do you think it would feel being the one paying the electric bills up until now?

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#37

Post by Randor »

kayo_20211030 wrote: post #30
Randor wrote: post #19

Earlier quoted context omitted.

The energy claims up to ~70% can be verified. The inference implementation is here: https://github.com/microsoft/BitNet

I'm not an AI person, in any technical sense. The savings being claimed, and I assume verified, are on ARM and x86 chips. The piece doesn't mention swapping mult to add, and a 1-bit LLM is, well, a 1-bit LLM. Also, > Additionally, it reduces energy consumption by 55.4% to 70.0% With humility, I don't know what that means. It seems like some dubious math with percentages.

> I don't know what that means. It seems like some dubious math with percentages.

I would start by downloading a 1.58 model such as: https://huggingface.co/HF1BitLLM/Llama3-8B-1.58-100B-tokens

Run the non-quantized version of the model on your 3090/4090 gpu and observe the power draw. Then load the 1.58 model and observe the power usage. Sure, the numbers have a wide range because there are many gpu/npu to make the comparison.

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#39

Post by stefan_ »

kayo_20211030 wrote: post #10

Extraordinary claims require extraordinary evidence. Maybe it's possible, but consider that some really smart people, in many different groups, have been working diligently in this space for quite a while; so claims of 95% savings on energy costs _with equivalent performance_ is in the extraordinary category. Of course, we'll see when the tide goes out.

I mean, all these smart people would rather pay NVIDIA all their money than make AMD viable. And yet they tell us its all MatMul.
Post reply on HN