Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI engineers claim new algorithm reduces AI power consumption by 95%

tomshardware.com

Post reply on HN
Print view View original

161–170 of 174 posts

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#161

Post by datavirtue »

recursive wrote: post #91
jimmaswell wrote: post #72

Earlier quoted context omitted.

They already have been. Even just in programming, even just Copilot has been a life changing productivity booster.

I've been using copilot for several months. If I could figure out a way to measure its impact on my productivity, I'd probably see a single digit percentage boost in "productivity". This is not life-changing for me. And for some tasks, it's actually worse than nothing. As in, I spend time feeding it a task, and it just completely fails to do anything useful.

If you are in maintenance mode your visits to Copilot will be rare. If you are building greenfield, use goes through the roof. All those test cases, nevermind all the POC and framework scaffolding and other boilerplate that is now completely unacceptable as a use of developer time.

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#162

Post by datavirtue »

namaria wrote: post #116
atq2119 wrote: post #110

Earlier quoted context omitted.

> I almost forgot how much it sucks having to jump to google every other minute Even allowing for some hyperbole, your programming experience is extremely different from mine. Looking anything up outside the IDE, let alone via Google, is by far the exception for me rather than the rule. I've long suspected that this kind of difference explains a lot of the difference in how Copilot is perceived.

Claiming LLMs are a massive boost for coding productivity is becoming a red flag that the claimant has a tenuous grasp on the skills necessary. Yeah if you have to look up everything all the time and you can't tell the AI slop isn't very good, you can put out code quite fast.

Nope, just want it to write tests and other low value work so I can get shit done. Some of it depends on the stakes of your job. Are you floating along day by day in big corp or are you grinding it out at a startup? Those working at the startup have to use coding assistants, period.

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#163

Post by mecsred »

crazygringo wrote: post #139
mecsred wrote: post #97

Earlier quoted context omitted.

If 20 mins of informations can legitimately be condensed into 20 seconds, it sounds like the original wasn't worth reading in the first place. Could have skipped the llm entirely.

> it sounds like the original wasn't worth reading in the first place But if that's the only place that contained the information you needed, then you have no choice. There's a lot of material out there that is badly written, badly organized, badly presented. LLM's can be a godsend for extracting the information you actually need without wasting 20 minutes wading through the muck.

Yeah I can see that use case, I just wouldn't trust an LLM to decide "is this worth reading". May as well flip a coin.

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#164

Post by throwawaymaths »

kayo_20211030 wrote: post #132
throwawaymaths wrote: post #66

Earlier quoted context omitted.

> If the 95% savings actually has legs some smart chip manufacturer will do the math and make the chips Terrible logic. By a similar logic we wouldn't be using python for machine learning at all, for example (or x86 for compute). Yet here we are.

What's wrong with the logic? A caveat in the paper is that the technique will save 95% energy but that the technique will not run efficiently on current chips. I'm saying that if the new technique needs new chips and saves 95% of energy costs with the same performance, someone will make the chips. I say nothing about how and why we do ML as we do today - the 100% energy usage level.

It's Terrible logic because it doesn't take into account the way this industry works. We don't do things because they are better. We do things because we can convince investors, because it's hirable, because we don't want to learn something new, because we're afraid our built up knowledge base is going to become obsolete, so we pull more people into our technical debt, etc.

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#165

Post by recursive »

datavirtue wrote: post #161
recursive wrote: post #91

Earlier quoted context omitted.

I've been using copilot for several months. If I could figure out a way to measure its impact on my productivity, I'd probably see a single digit percentage boost in "productivity". This is not life-changing for me. And for some tasks, it's actually worse than nothing. As in, I spend time feeding it a task, and it just completely fails to do anything useful.

If you are in maintenance mode your visits to Copilot will be rare. If you are building greenfield, use goes through the roof. All those test cases, nevermind all the POC and framework scaffolding and other boilerplate that is now completely unacceptable as a use of developer time.

I'm building "greenfield". I still use it at least daily, but the benefit just struggles to outweigh the cost of invoking it. Maybe I don't understand how to use it.

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#166

Post by datavirtue »

recursive wrote: post #165
datavirtue wrote: post #161

Earlier quoted context omitted.

If you are in maintenance mode your visits to Copilot will be rare. If you are building greenfield, use goes through the roof. All those test cases, nevermind all the POC and framework scaffolding and other boilerplate that is now completely unacceptable as a use of developer time.

I'm building "greenfield". I still use it at least daily, but the benefit just struggles to outweigh the cost of invoking it. Maybe I don't understand how to use it.

It really depends on what you are doing and what tech you are using. I use it to teach me or build out ideas quickly or solve for complex issues. Mostly these days I use it as a memory aid or to bounce ideas off it. In my job I have to move quickly and stay focused as I'm driving improvements to a tech stack that reaches across four verticals, each having their own quirks and tech stacks. It's great for jogging my memory and helping to flush out ideas and approaches that I then bounce off the dev teams. Super helpful.

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#167

Post by Sohcahtoa82 »

3abiton wrote: post #111
crazygringo wrote: post #20

Earlier quoted context omitted.

No. But it does potentially mean that either current or future-tweaked GPUs could run a lot more efficiently -- meaning much faster or with much less energy consumption. You still need the GPU parallelism though.

This is still amazing work, imagine running chungus models on a single 3090.

The bottleneck on a consumer-grade GPU like a 3090 isn't the processing power, it's the lack of RAM. The PCI-Express bus ends up being your bottleneck from having to swap in parts of the model.

Even with PCIe 5.0 and 16 lanes, you only get 64 GB/s of bandwidth. If you're trying to run a model too big for your GPU, then for every token, it has to reload the entire model. With a 70B parameter model, 8 bit quantization, you're looking at just under 1 token/sec just from having to transfer parts of the model in constantly. Making the actual computation faster won't make it any faster.

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#168

Post by dragonwriter »

Sohcahtoa82 wrote: post #167
3abiton wrote: post #111

Earlier quoted context omitted.

This is still amazing work, imagine running chungus models on a single 3090.

The bottleneck on a consumer-grade GPU like a 3090 isn't the processing power, it's the lack of RAM. The PCI-Express bus ends up being your bottleneck from having to swap in parts of the model. Even with PCIe 5.0 and 16 lanes, you only get 64 GB/s of bandwidth. If you're trying to run a model too big for your GPU, then for every token, it has to reload the entire model. With a 70B parameter model, 8 bit quantization,…

OTOH, doesn't it also mean that (given appropriate software framework support) iGPUs with less processing capacity and slower-but-more RAM available (because system RAM is comparatively cheap and plentiful compared to VRAM) without swapping anything are more competitive against consumer dGPUs with fast-but-small RAM for both inference and training with larger models?

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#169

Post by Sohcahtoa82 »

dragonwriter wrote: post #168
Sohcahtoa82 wrote: post #167

Earlier quoted context omitted.

The bottleneck on a consumer-grade GPU like a 3090 isn't the processing power, it's the lack of RAM. The PCI-Express bus ends up being your bottleneck from having to swap in parts of the model. Even with PCIe 5.0 and 16 lanes, you only get 64 GB/s of bandwidth. If you're trying to run a model too big for your GPU, then for every token, it has to reload the entire model. With a 70B parameter model, 8 bit quantization,…

OTOH, doesn't it also mean that (given appropriate software framework support) iGPUs with less processing capacity and slower-but-more RAM available (because system RAM is comparatively cheap and plentiful compared to VRAM) without swapping anything are more competitive against consumer dGPUs with fast-but-small RAM for both inference and training with larger models?

System memory isn't that fast, either. Even with DDR5-8400, the fastest memory you can get right now, you're only looking at a memory transfer speed of 67.2 GB/s, barely faster than the PCI-E bus. So even if you could store that entire 70B model in RAM, you're still getting just under 1 token/sec, and that's assuming your CPU doesn't become a bottleneck.

Your best bet would likely be a laptop that has integrated system RAM with VRAM, but I don't think any of those offer enough RAM to store an entire 70B model. A 7B parameter model would work fine, but you could do those on a consumer-grade GPU anyways.

Re: AI engineers claim new algorithm reduces AI power consumption by 95%

#170

Post by DeveloperErrata »

Sohcahtoa82 wrote: post #169
dragonwriter wrote: post #168

Earlier quoted context omitted.

OTOH, doesn't it also mean that (given appropriate software framework support) iGPUs with less processing capacity and slower-but-more RAM available (because system RAM is comparatively cheap and plentiful compared to VRAM) without swapping anything are more competitive against consumer dGPUs with fast-but-small RAM for both inference and training with larger models?

System memory isn't that fast, either. Even with DDR5-8400, the fastest memory you can get right now, you're only looking at a memory transfer speed of 67.2 GB/s, barely faster than the PCI-E bus. So even if you could store that entire 70B model in RAM, you're still getting just under 1 token/sec, and that's assuming your CPU doesn't become a bottleneck. Your best bet would likely be a laptop that has integrated syst…

Macbook Pros with M3 & integrated RAM & VRAM can do 70B models :)
Post reply on HN