Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

arxiv.org

Post reply on HN
Print view View original

31–40 of 59 posts

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#31

Post by stymaar »

rpozarickij wrote: post #17
jmiskovic wrote: post #9

Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…

> During the winter time the GPU also doubles as a 300W in-house heater There could be a service that works in reverse where if someone needs a heater for a few months, they could rent a portable server (e.g. using older repurposed GPUs) with a built-in 5G modem that would run inference on LLM queries. As an incentive perhaps renting itself could be free (or you could earn money?), but you'd still have to pay your el…

I saw a company pitching almost exactly this on linkedin earlier this month. Except not as a portable heater but as your house's central heating system.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#32

Post by stymaar »

iovrthoughtthis wrote: post #25
jmiskovic wrote: post #9

Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…

What devices and models are people running locally?

Qwen 3.6-35B-A3B is surprisingly OK on a medium-range business laptop (Lenovo).

A bit slow for agentic coding of course but fine for any chatbot use-case.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#33

Post by mattmanser »

stymaar wrote: post #30
rckoepke wrote: post #20

Earlier quoted context omitted.

> During the winter time the GPU also doubles as a 300W in-house heater. Just a reminder that heat pumps can consume 300W of electricity to provide 1200W of heat.

Of course, but the power dissipated in a data center provide zero heat for your house. (Tangent: I wonder why we haven't seen deployment of organic Rankin cycle generators in AI data centers, the exhaust temperature should be compatible and that could yield a 10-20% energy bill saving).

I have no idea what they use, but Stockholm is already using the waste heat from data centres:

https://eu-mayors.ec.europa.eu/en/news/stockholm-sweden-heat...

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#34

Post by qtqtqt »

iovrthoughtthis wrote: post #25
jmiskovic wrote: post #9

Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…

What devices and models are people running locally?

Any good models that do not need a dedicated GPU?

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#35

Post by stymaar »

mattmanser wrote: post #33
stymaar wrote: post #30

Earlier quoted context omitted.

Of course, but the power dissipated in a data center provide zero heat for your house. (Tangent: I wonder why we haven't seen deployment of organic Rankin cycle generators in AI data centers, the exhaust temperature should be compatible and that could yield a 10-20% energy bill saving).

I have no idea what they use, but Stockholm is already using the waste heat from data centres: https://eu-mayors.ec.europa.eu/en/news/stockholm-sweden-heat...

They use it in the city heating system, which is obviously the optimal way of reusing the heat but most data centers aren't put next to such a heating system.

I'm talking about making electricity back from the heat (using a low-temp thermodynamic cycle). It has a low yield (due to the low input temperature) but it's usually economically viable when using heat that would end up in the heavens anyway.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#36

Post by lioeters »

iLoveOncall wrote: post #18
utopiah wrote: post #3

Earlier quoted context omitted.

I don't think that's what they are saying. In fact if they did the metric would be pointless. Rather they are saying by estimating that value on different architectures, one can find more efficient ones. They use open model to be able to remove unknowns. They aren't advocating for one model or another, only more efficient architectures.

Then the metric should be Watts per Intelligence, not the contrary.

Is there any practical difference? One is just an inverse of the other and can be trivially derived.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#37

Post by WASDx »

I tried calculating historical "intelligence per cost" recently but stopped when I realized intelligence is not linear. For any meaningful "x per y" you can just double "y" if you have a half as efficient system to get the same result but so-called intelligence doesn't work like that.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#38

Post by jonas21 »

stymaar wrote: post #30
rckoepke wrote: post #20

Earlier quoted context omitted.

> During the winter time the GPU also doubles as a 300W in-house heater. Just a reminder that heat pumps can consume 300W of electricity to provide 1200W of heat.

Of course, but the power dissipated in a data center provide zero heat for your house. (Tangent: I wonder why we haven't seen deployment of organic Rankin cycle generators in AI data centers, the exhaust temperature should be compatible and that could yield a 10-20% energy bill saving).

That's also true in the summer when you don't want that heat dissipated in your house.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#39

Post by MisterTea »

stymaar wrote: post #30
rckoepke wrote: post #20

Earlier quoted context omitted.

> During the winter time the GPU also doubles as a 300W in-house heater. Just a reminder that heat pumps can consume 300W of electricity to provide 1200W of heat.

Of course, but the power dissipated in a data center provide zero heat for your house. (Tangent: I wonder why we haven't seen deployment of organic Rankin cycle generators in AI data centers, the exhaust temperature should be compatible and that could yield a 10-20% energy bill saving).

> I wonder why we haven't seen deployment of organic Rankin cycle generators in AI data centers

Because there was never a long term plan for AI data centers. It's an AI market capture and cash grab scheme that ends when local models eat their lunch.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#40

Post by RunSet »

surprisetalk wrote: post #21

I rather like this paper, but I think it is generous to say their benchmarks measure "intelligence". [0] https://arxiv.org/pdf/1911.01547 We have no dang clue what intelligence is, nor how to measure it. [1] https://taylor.town/crowpower

"We see a future where intelligence is a utility like electricity or water and people buy it from us on a meter"

- Sam Altman[0]

"Can you define intelligence?"

"Yes, it is this many moneys."

[0] https://www.businessinsider.com/sam-altman-ai-utility-electr...

Post reply on HN