Saw some measurements on SBC NPUs (3588) and that did seem to have a decent win on power over CPU...but also a perplexity loss relative to CPU so think this will prove quite hard to reliably quantify in practice.
Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
11–20 of 62 posts
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#12Earlier quoted context omitted.
I don't think that's what they are saying. In fact if they did the metric would be pointless. Rather they are saying by estimating that value on different architectures, one can find more efficient ones. They use open model to be able to remove unknowns. They aren't advocating for one model or another, only more efficient architectures.
Intelligence per Joule would be more appropriate in many cases. If a model can do the same work but takes 10 times as long as a bigger one that can still be useful (e.g. due to memory constraints), but at the same wattage it burns 10 times the energy. Even more so on mobile devices.
They also define and measure an "IPJ" as well as "IPW"
> the NVIDIA B200 achieves 1.6× to 2.3× higher intelligence per joule than the APPLE M4 MAX across QWEN 3 and GPT-OSS model variants
The B200 = "cloud", M4 = "local".
So "cloud" does even better in energy than it does in power compared to "local". Or, to flip it, "local" is both slower and more expensive than "cloud".
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#13Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…
> It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands.
This is only true if your local model is already resident in RAM / VRAM.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#14> We propose intelligence per watt (IPW), task accuracy per unit of power Stupid metric. It's not because a model is better performing that it necessarily requires more energy or compute.
Not a well-thought-out take.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#15Very cool paper and some interesting things for local serving. Though its a little apples-to-oranges we do publish live energy stats for all models on our service here https://portal.neuralwatt.com/energy-pricing in case you are interested in what this looks like on the cloud side. FWIW DSV4.1 flash is really getting popular due to its IPW.
Some of the items like model routing, if you do it per request instead of per session, can break down on the cloud from an energy and cost POV since one of the best things you can do for both is to maintain the KV cache which both reduces time component of energy and the quite expensive prefill energy.
I am keen on the future where we have local/cloud hybrid serving which is cache aware. I do think that could be the best use of energy resources for AI.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#16[dead]
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#17Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…
> During the winter time the GPU also doubles as a 300W in-house heater
There could be a service that works in reverse where if someone needs a heater for a few months, they could rent a portable server (e.g. using older repurposed GPUs) with a built-in 5G modem that would run inference on LLM queries. As an incentive perhaps renting itself could be free (or you could earn money?), but you'd still have to pay your electricity bill.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#18> We propose intelligence per watt (IPW), task accuracy per unit of power Stupid metric. It's not because a model is better performing that it necessarily requires more energy or compute.
I don't think that's what they are saying. In fact if they did the metric would be pointless. Rather they are saying by estimating that value on different architectures, one can find more efficient ones. They use open model to be able to remove unknowns. They aren't advocating for one model or another, only more efficient architectures.
Then the metric should be Watts per Intelligence, not the contrary.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#19Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…
> It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. This is only true if your local model is already resident in RAM / VRAM.
In my case the bad internet also tips the scale.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#20Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…
> During the winter time the GPU also doubles as a 300W in-house heater.
Just a reminder that heat pumps can consume 300W of electricity to provide 1200W of heat.