[dead]
Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
41–50 of 62 posts
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#42Did I miss something or does this article not bother to indicate how much RAM their M4-Max had?
[dead]
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#43Earlier quoted context omitted.
Then the metric should be Watts per Intelligence, not the contrary.
Is there any practical difference? One is just an inverse of the other and can be trivially derived.
Yes, there is a massive difference, and you cannot invert them to get the other.
Watt per Intelligence means that you have a fixed, deterministic, measure of intelligence, and you calculate how many watts it takes to get there.
If your goal is to measure which model can reach a specific outcome with the least energy possible (which is what GP says the goal is for this metric), then you cannot have a variable outcome, which is what intelligence per watt describes. As opposed to watt per intelligence, where the outcome is fixed and the numerator defines how much energy expenditure is needed to reach this fixed outcome.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#44Earlier quoted context omitted.
Of course, but the power dissipated in a data center provide zero heat for your house. (Tangent: I wonder why we haven't seen deployment of organic Rankin cycle generators in AI data centers, the exhaust temperature should be compatible and that could yield a 10-20% energy bill saving).
> I wonder why we haven't seen deployment of organic Rankin cycle generators in AI data centers Because there was never a long term plan for AI data centers. It's an AI market capture and cash grab scheme that ends when local models eat their lunch.
That makes no sense. Do you think their plan is to spend billions of dollars and build infrastructure mega projects just to let it sit idle?
Did the rise personal computing make data centers and super computers obsolete?
Any advance in inference that allows local models to do the job will also benefit hyperscalers. Imagine the sheer amount of compute they could throw at problems if each 32GB of VRAM was enough for frontier reasoning.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#45[flagged]
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#46Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…
What devices and models are people running locally?
A MacBook Pro is probably the best bang for your buck. I have a 64gb m4 max, but you’ll run decent models with half that ram I think.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#47Earlier quoted context omitted.
Of course, but the power dissipated in a data center provide zero heat for your house. (Tangent: I wonder why we haven't seen deployment of organic Rankin cycle generators in AI data centers, the exhaust temperature should be compatible and that could yield a 10-20% energy bill saving).
That's also true in the summer when you don't want that heat dissipated in your house.
Easier said than done, but if you water-cool the GPU just upstream of your domestic water heater, it wouldn't be a bad thing. Perhaps using coolant and a counterflow heat exchanger, rather than the potable water, but the point stands. Would just need a secondary tank as a buffer (able to soak up heat from the GPU at all times, even when there's no demand for hot water) which then flushes in when demand exists.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#48Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…
> We've reached the point where local LLMs are good enough!
For some tasks, yes. For most of my deeper work they're not even close to my subscriptions.
> It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands.
I have some decent LLM hardware here and I strongly disagree with this. Claude responds quickly. Using Fable or Opus it will deliver a working result faster than my local models because it gets there in fewer tokens. That's just how it is.
> During the winter time the GPU also doubles as a 300W in-house heater.
This is a curse in the summer. I'm feeling it right now.
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#49Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…
> During the winter time the GPU also doubles as a 300W in-house heater There could be a service that works in reverse where if someone needs a heater for a few months, they could rent a portable server (e.g. using older repurposed GPUs) with a built-in 5G modem that would run inference on LLM queries. As an incentive perhaps renting itself could be free (or you could earn money?), but you'd still have to pay your el…
> you'd still have to pay your electricity bill.
I should only have to pay 1/3 of what it adds to my bill, given that it's 1/3 as efficient as a heat pump. And that coefficient is to be adjusted as outside temp changes (assuming air source heat pump; water source would have a stable COP).
Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI
#50Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…
This is annoying but, can you point me to where to get started with local models? A repo, website, something?