I don't think algorithms will change energy consumption. There is always max capacity needed in terms of computing. If tomorrow a new algorithm increases the performance 4 times, we will just have 4 times more computing.
AI engineers claim new algorithm reduces AI power consumption by 95%
61–70 of 174 posts
Re: AI engineers claim new algorithm reduces AI power consumption by 95%
#62Maybe I am just a natural skeptic, but whenever I see a headline that says 'method x reduces y by z%'; but when you read the text it instead says that optimizing some step 'could potentially reduce y by up to z%'; I am suspicious. Why not publish some actual benchmarks that prove your claim in even a few special cases?
Because as disappointing as modern life is, you need clickbait headlines to drive traffic. You did the right thing by reading the article though, that's where the information is, not the title.
Fair enough, but then I want a way to penalize publishers for abusing clickbait. There is no "unread" button, and there is no way to unsubscribe to advertisement-based sites.
Even on sites that have a "Like / Don't like" button, my understanding is that clicking "Don't like" is a form of "engagement", that the suggestion algorithm are going to reward.
Give me a button that says "this article was a scam", and have the publisher give the advertisement money back. Of better yet, give the advertisement money to charity / public services / whatever.
Take a cut of the money being transfered, charge the publishers for being able to get a "clickbait free" green mark if they implement the scheme.
Track the kind of articles that generate the most clickbait-angry comment. Sell back the data.
There might a business model.
Re: AI engineers claim new algorithm reduces AI power consumption by 95%
#63Maybe I am just a natural skeptic, but whenever I see a headline that says 'method x reduces y by z%'; but when you read the text it instead says that optimizing some step 'could potentially reduce y by up to z%'; I am suspicious. Why not publish some actual benchmarks that prove your claim in even a few special cases?
Well, one, because the headline isn't from the researchers, its from a popular press report (not even the one posted here, originally, this is secondary reporting of another popular press piece) and isn't what the paper claims so it would be odd for the paper's authors to conduct benchmarks to justify it. (And, no, even the "up to 95%" isn't from the paper, the cost savings are cited per operation depending on operation and the precision the operation is conducted at, are as high as 97.3%, are based on research already done establishing the energy cost of math operations on modern compute hardware, but no end-to-end cost savings claim is made.)
And, two, because the actual energy cost savings claimed aren't even the experimental question -- the energy cost differences between various operations on modern hardware have been established in other research, the experimental issue here was whether the mathematical technique that enables using the lower energy cost operations performs competitively on output quality with existing implementations when substituted in for LLM inference.
Re: AI engineers claim new algorithm reduces AI power consumption by 95%
#64Earlier quoted context omitted.
Because as disappointing as modern life is, you need clickbait headlines to drive traffic. You did the right thing by reading the article though, that's where the information is, not the title.
Fair enough, but then I want a way to penalize publishers for abusing clickbait. There is no "unread" button, and there is no way to unsubscribe to advertisement-based sites. Even on sites that have a "Like / Don't like" button, my understanding is that clicking "Don't like" is a form of "engagement", that the suggestion algorithm are going to reward. Give me a button that says "this article was a scam", and have the…
I doubt there’s a business model there because who is going to opt in to a scheme that loses them money?
What could work is social media giving people an easy button to block links to specific websites from appearing in their feed, or something along those lines. It’s a nice user feature, and having every clickbait article be a chance someone will choose to never see your website again could actually reign in some of the nonsense.
Re: AI engineers claim new algorithm reduces AI power consumption by 95%
#65Earlier quoted context omitted.
I had a feeling it had to be something like massive waste due to a misguided feature of the algorithms that shouldn't have been there in the first place. Once the "math is done" quite likely it would have paid off better than most investments for the top people to have spent a few short years working with grossly underpowered hardware until they could come up with amazing results there before scaling up. Rather than…
Isn’t this paper pretty much about spending a few short years to improve the performance? Or are you arguing that the same people who made breakthroughs over the last few years should have also done the optimization work?
>the same people who made breakthroughs over the last few years should have also done the optimization work
I never thought it would be ideal if it was otherwise, so I guess so.
When I first considered neural nets from state-of-the art vendors to assist with some non-linguistic situations over 30 years ago, it wasn't quite ready for prime time and I could accept that.
I just don't have generic situations all the time which would benefit me, so it's clearly my problems that have the deficiencies ;\
What's being done now with all the resources being thrown at it is highly impressive, and gaining all the time, no doubt about it. It's nice to know there are people that can afford it.
I truly look forward to more progress, and this may be the previously unreached milestone I have been detecting that might be a big one.
Still not good enough for what I need yet so far though. And I can accept that as easily as ever.
That's why I put up my estimation that not all of those 30+ years has been spent without agonizing over something ;)
Re: AI engineers claim new algorithm reduces AI power consumption by 95%
#66Extraordinary claims require extraordinary evidence. Maybe it's possible, but consider that some really smart people, in many different groups, have been working diligently in this space for quite a while; so claims of 95% savings on energy costs _with equivalent performance_ is in the extraordinary category. Of course, we'll see when the tide goes out.
re: all above/below comments. It's still an extraordinary claim. I'm not claiming it's not possible, nor am I claiming that it's not true, or, at least, honest. But, there will need to be evidence that using real machines, and using real energy an _equivalent performance_ is achievable. A defense that "there are no suitable chips" is a bit disingenuous. If the 95% savings actually has legs some smart chip manufacture…
> If the 95% savings actually has legs some smart chip manufacturer will do the math and make the chips
Terrible logic. By a similar logic we wouldn't be using python for machine learning at all, for example (or x86 for compute). Yet here we are.
Re: AI engineers claim new algorithm reduces AI power consumption by 95%
#67Maybe I am just a natural skeptic, but whenever I see a headline that says 'method x reduces y by z%'; but when you read the text it instead says that optimizing some step 'could potentially reduce y by up to z%'; I am suspicious. Why not publish some actual benchmarks that prove your claim in even a few special cases?
https://github.com/microsoft/BitNet
"The first release of bitnet.cpp is to support inference on CPUs. bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by 55.4% to 70.0%, further boosting overall efficiency. On x86 CPUs, speedups range from 2.37x to 6.17x with energy reductions between 71.9% to 82.2%. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. More details will be provided soon."
Re: AI engineers claim new algorithm reduces AI power consumption by 95%
#68Original discussion of the preprint: https://news.ycombinator.com/item?id=41784591
Ahh, there it is! I was sure we had discussed this paper before.
Re: AI engineers claim new algorithm reduces AI power consumption by 95%
#69Maybe I am just a natural skeptic, but whenever I see a headline that says 'method x reduces y by z%'; but when you read the text it instead says that optimizing some step 'could potentially reduce y by up to z%'; I am suspicious. Why not publish some actual benchmarks that prove your claim in even a few special cases?
https://github.com/microsoft/BitNet "The first release of bitnet.cpp is to support inference on CPUs. bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by 55.4% to 70.0%, further boosting overall efficiency. On x86 CPUs, speedups range from 2.37x to 6.17x with energy reductions between 71.9% to 82.2%. Furt…
Damn. Seems almost too good to be true. Let’s see where this goes in two weeks.
Re: AI engineers claim new algorithm reduces AI power consumption by 95%
#70Earlier quoted context omitted.
I had a feeling it had to be something like massive waste due to a misguided feature of the algorithms that shouldn't have been there in the first place. Once the "math is done" quite likely it would have paid off better than most investments for the top people to have spent a few short years working with grossly underpowered hardware until they could come up with amazing results there before scaling up. Rather than…
Faster progress was absolutely worth it. Spending years agonizing over theory to save a bit of electric would have been a massive disservice to the world.
You're sort of presuming that LLMs are going to be a massive service to the world there, aren't you? I think the jury is still out on that one.