Intelligence per Watt: Measuring Intelligence Efficiency of Local AI • Hacker News Board

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#61

Post by fc417fc802 » Thu, Sep 17, 2026, 12:20 AM UTC

tccole wrote: post #24

jmiskovic wrote: post #9

Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…

For sure but aren’t there more regulations and building codes around heaters? It’s been close to a decade since I took heat and mass transfer courses but this seems like saying you can use your oven as a heater for your house in the wintertime. Please correct me if I am wrong.

Yes it is exactly like that. And I've done that before when my central heat went out so I'm not sure what you're trying to say here?

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#62

Post by fc417fc802 » Thu, Sep 17, 2026, 12:27 AM UTC

frumiousirc wrote: post #10

api wrote: post #5

Unless I misread it, are they saying local GPUs use less energy? That’s surprising, almost unbelievable, due to batching. Local is usually not batched.

> Unless I misread it, are they saying local GPUs use less energy? You misread it. From the abstract: > local accelerators achieve at least 1.4× lower IPW than cloud accelerators running identical models That's "intelligence per watt". They also have IPJ, per Joule. So, they find local is 40% "dumber" than cloud for the same power or 40% more power for the same "intelligence". Tables 13 and 14 summarize their IPW and…

> Of course a local PC is used for other things and the idleness penalty must somehow account for that.

I'd argue the idleness penalty only applies if you wouldn't have otherwise had equivalent hardware. If you already have the exact same dGPU for gaming and it doubles up for inference then the only inefficiency is power consumption (both by the GPU and potentially by AC for your living space).

Conversely I think we should consider that privacy, distributed compute that you can access without an intermediary, and a more distributed power grid all provide net benefits to society at large.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#63

Post by thin_carapace » Thu, Sep 17, 2026, 1:25 AM UTC

01284a7e wrote: post #50

jmiskovic wrote: post #9

Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…

This is annoying but, can you point me to where to get started with local models? A repo, website, something?

visit canirun.ai to find a suitable model, download .gguf of a version of that model from huggingface, plug .gguf into ollama