I rather like this paper, but I think it is generous to say their benchmarks measure "intelligence".
[0] https://arxiv.org/pdf/1911.01547
We have no dang clue what intelligence is, nor how to measure it.
Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
21–30 of 62 posts
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#22Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…
> During the winter time the GPU also doubles as a 300W in-house heater. Just a reminder that heat pumps can consume 300W of electricity to provide 1200W of heat.
Under ideal conditions with outside temperatures*
But yes
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#23Unless I misread it, are they saying local GPUs use less energy? That’s surprising, almost unbelievable, due to batching. Local is usually not batched.
You can most definitely batch local models and do unattended inference on a 24/7 basis to maximize utilization on local hardware too. The limits are usually set by some combination of memory utilization for KV cache (particularly on small dGPUs) and overall thermals/power limits (particularly on iGPUs with unified RAM/VRAM). (If you're not near thermal limits, the main alternative to batching is to use MTP or speculative decoding in order to raise arithmetic intensity and speed with the same memory utilization. But batching requests is generally viewed as preferable.) Newer models, especially from DeepSeek, do a nice job of reducing KV cache memory impact for any given context length and/or amount of parallel sessions, so batching on local hw really ought to be quite feasible.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#24Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…
For sure but aren’t there more regulations and building codes around heaters? It’s been close to a decade since I took heat and mass transfer courses but this seems like saying you can use your oven as a heater for your house in the wintertime.
Please correct me if I am wrong.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#25Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…
What devices and models are people running locally?
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#26ok now let's do some math to calculate the current levels of daily intelligence to calculate how much power we need to replace 10% of it with AI. Never mind new use cases or approaching human replacement. My guess is it is a few orders of magnitude more power than we produce today.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#27Did I miss something or does this article not bother to indicate how much RAM their M4-Max had?
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#28Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…
> During the winter time the GPU also doubles as a 300W in-house heater There could be a service that works in reverse where if someone needs a heater for a few months, they could rent a portable server (e.g. using older repurposed GPUs) with a built-in 5G modem that would run inference on LLM queries. As an incentive perhaps renting itself could be free (or you could earn money?), but you'd still have to pay your el…
I remember a company that sold or used bitcoin mining rigs as swimming pool heaters.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#29I rather like this paper, but I think it is generous to say their benchmarks measure "intelligence". [0] https://arxiv.org/pdf/1911.01547 We have no dang clue what intelligence is, nor how to measure it. [1] https://taylor.town/crowpower
it really does feel like 'intelligence' is becoming so much more of a marketing term moreso than it is any actual definitive measure. I was acquaintances with a cognitive science post-doc many years ago who trained under the same program that Douglas Hofstadter was part of (and who he saw as a great popular science communicator but perhaps a bit over-rated when it came to his own research)
my very naive question to him back then was how close we were to understanding human cognition. we were both fans of grand strategy games (though the few hundred hours of Stellaris I played vs his thousands in EU4 paled in comparison) and I was asking if it was possible to map human cognition to the same array of interdependent logical chains-of-reasoning that games like that could be boiled down to
his answer, in short, was 'we are so, so, so far away and no, that's, at best, a reductive mental model of intelligence'
I keep that conversation in mind whenever I hear about all this talk of AGI - that realistically we're so far away from actual AGI in the same way that the inventor(s) of the wheel were from a gas-powered car, and there's many paradigm shifts to go in how we even understand what the nature of intelligence is before we get there
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#30Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…
> During the winter time the GPU also doubles as a 300W in-house heater. Just a reminder that heat pumps can consume 300W of electricity to provide 1200W of heat.
Of course, but the power dissipated in a data center provide zero heat for your house.
(Tangent: I wonder why we haven't seen deployment of organic Rankin cycle generators in AI data centers, the exhaust temperature should be compatible and that could yield a 10-20% energy bill saving).