Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

arxiv.org

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#51

Post by adgjlsfhk1 »

stymaar wrote: post #35
mattmanser wrote: post #33

Earlier quoted context omitted.

I have no idea what they use, but Stockholm is already using the waste heat from data centres: https://eu-mayors.ec.europa.eu/en/news/stockholm-sweden-heat...

They use it in the city heating system, which is obviously the optimal way of reusing the heat but most data centers aren't put next to such a heating system. I'm talking about making electricity back from the heat (using a low-temp thermodynamic cycle). It has a low yield (due to the low input temperature) but it's usually economically viable when using heat that would end up in the heavens anyway.

the heat is way too low grade. typical water cooling setups only have ~50C water. that's barely anything

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#52

Post by lowbloodsugar »

01284a7e wrote: post #50
jmiskovic wrote: post #9

Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…

This is annoying but, can you point me to where to get started with local models? A repo, website, something?

Ask Claude.ai. Tell it about your hardware. What you plan on doing. Can start with chat. Then ask it how to install Claude remote control on you system. Then it can work while you sleep. Now I have a solid local model that I use instead of Claude.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#53

Post by perfectbeeing »

drivebyhooting wrote: post #44
MisterTea wrote: post #39

Earlier quoted context omitted.

> I wonder why we haven't seen deployment of organic Rankin cycle generators in AI data centers Because there was never a long term plan for AI data centers. It's an AI market capture and cash grab scheme that ends when local models eat their lunch.

That makes no sense. Do you think their plan is to spend billions of dollars and build infrastructure mega projects just to let it sit idle? Did the rise personal computing make data centers and super computers obsolete? Any advance in inference that allows local models to do the job will also benefit hyperscalers. Imagine the sheer amount of compute they could throw at problems if each 32GB of VRAM was enough for fr…

CCC was building the Cray-3/SSS when it went into Chapter 11 bankruptcy in March 1995.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#54

Post by lioeters »

iLoveOncall wrote: post #43
lioeters wrote: post #36

Earlier quoted context omitted.

Is there any practical difference? One is just an inverse of the other and can be trivially derived.

Yes, there is a massive difference, and you cannot invert them to get the other. Watt per Intelligence means that you have a fixed, deterministic, measure of intelligence, and you calculate how many watts it takes to get there. If your goal is to measure which model can reach a specific outcome with the least energy possible (which is what GP says the goal is for this metric), then you cannot have a variable outcome,…

Here is a more everyday example: distance traveled by a vehicle and the amount of fuel consumed.

> Fuel efficiency can be expressed in terms of the volume of fuel to travel a given distance, such as in litres per 100 kilometres, or through its inverse, the distance traveled per unit volume of fuel consumed, as in kilometres per litre.

If you have a car that consumes 1 liter of fuel per 100 kilometers, it's the same as saying it travels 1 kilometer per 0.01 liter of fuel. They are equivalent, describing the same relationship and proportion. It makes no difference whether the quantity is a "deterministic" or "variable" measurement. You can use either unit, depending on the aim of your calculation.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#55

Post by zrail »

hunter2_ wrote: post #47
jonas21 wrote: post #38

Earlier quoted context omitted.

That's also true in the summer when you don't want that heat dissipated in your house.

Easier said than done, but if you water-cool the GPU just upstream of your domestic water heater, it wouldn't be a bad thing. Perhaps using coolant and a counterflow heat exchanger, rather than the potable water, but the point stands. Would just need a secondary tank as a buffer (able to soak up heat from the GPU at all times, even when there's no demand for hot water) which then flushes in when demand exists.

Let's say your GPU uses a constant 300W. It takes approximately 1.16 Wh to heat a liter of water 1 degree celsius under 100% efficiency. This setup won't be 100% efficient, so let's round up to 1.5 Wh per liter per degree (roughly 80% efficient). A typical domestic water heater in the US 40-50 gallons, so roughly 160 liters.

Let's say your incoming water temperature is 18C and you want it preheated to 50C, which is 32C degree differential, which means you'll need 32 * 1.5 * 160 = 7680 Wh, or 25 hours straight to heat the buffer tank from scratch.

You'll need to purchase a small water-to-water heat exchanger ($50), two pumps ($100 each), a power supply for said pumps, hose and/or copper pipe and fittings, and various other sundries, plus the cost of a buffer tank ($600ish), so figure all in roughly $1000, plus the cost of electricity to run the pumps.

At $0.22/kWh you're saving roughly $450 a year with this setup in foregone water heating, but because it's not 100% efficient you're spending $500 in electricity to run your GPU 24/7/365, and that's the maximum you can possibly save with the above assumptions. Scale up for more GPUs and down accordingly for less usage as you see fit.

Alternatively, use air as the heat conductor by placing the GPU laden machine in the same space as a hybrid heat pump water heater.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#56

Post by teamonkey »

HPsquared wrote: post #28
rpozarickij wrote: post #17

Earlier quoted context omitted.

> During the winter time the GPU also doubles as a 300W in-house heater There could be a service that works in reverse where if someone needs a heater for a few months, they could rent a portable server (e.g. using older repurposed GPUs) with a built-in 5G modem that would run inference on LLM queries. As an incentive perhaps renting itself could be free (or you could earn money?), but you'd still have to pay your el…

I remember a company that sold or used bitcoin mining rigs as swimming pool heaters.

There was one that tried to sell them as free heating for the elderly.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#57

Post by AngryData »

drivebyhooting wrote: post #44
MisterTea wrote: post #39

Earlier quoted context omitted.

> I wonder why we haven't seen deployment of organic Rankin cycle generators in AI data centers Because there was never a long term plan for AI data centers. It's an AI market capture and cash grab scheme that ends when local models eat their lunch.

That makes no sense. Do you think their plan is to spend billions of dollars and build infrastructure mega projects just to let it sit idle? Did the rise personal computing make data centers and super computers obsolete? Any advance in inference that allows local models to do the job will also benefit hyperscalers. Imagine the sheer amount of compute they could throw at problems if each 32GB of VRAM was enough for fr…

They could also be betting that there will only be 1 or 2 big winners. And if they don't win they are saving on expenses, if they do win then they will have all the money they need to upgrade later if they want.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#58

Post by iLoveOncall »

lioeters wrote: post #54
iLoveOncall wrote: post #43

Earlier quoted context omitted.

Yes, there is a massive difference, and you cannot invert them to get the other. Watt per Intelligence means that you have a fixed, deterministic, measure of intelligence, and you calculate how many watts it takes to get there. If your goal is to measure which model can reach a specific outcome with the least energy possible (which is what GP says the goal is for this metric), then you cannot have a variable outcome,…

Here is a more everyday example: distance traveled by a vehicle and the amount of fuel consumed. > Fuel efficiency can be expressed in terms of the volume of fuel to travel a given distance, such as in litres per 100 kilometres, or through its inverse, the distance traveled per unit volume of fuel consumed, as in kilometres per litre. If you have a car that consumes 1 liter of fuel per 100 kilometers, it's the same a…

This has absolutely nothing to do with the problem at end. I'm so tired of people bringing up random metaphors that don't apply to justify obvious bullshit when it comes to AI.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#59

Post by applicative »

jmiskovic wrote: post #9

Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winte…

Local LLMs make my laptop hot and are wrong about everything.

Re: Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

#60

Post by fc417fc802 »

zrail wrote: post #55
hunter2_ wrote: post #47

Earlier quoted context omitted.

Easier said than done, but if you water-cool the GPU just upstream of your domestic water heater, it wouldn't be a bad thing. Perhaps using coolant and a counterflow heat exchanger, rather than the potable water, but the point stands. Would just need a secondary tank as a buffer (able to soak up heat from the GPU at all times, even when there's no demand for hot water) which then flushes in when demand exists.

Let's say your GPU uses a constant 300W. It takes approximately 1.16 Wh to heat a liter of water 1 degree celsius under 100% efficiency. This setup won't be 100% efficient, so let's round up to 1.5 Wh per liter per degree (roughly 80% efficient). A typical domestic water heater in the US 40-50 gallons, so roughly 160 liters. Let's say your incoming water temperature is 18C and you want it preheated to 50C, which is 3…

Just recycle an old water heater as the tank? I think your cost estimate is way overblown but it would certainly eat up a lot of time and effort to DIY. Regardless there's a much bigger flaw with this plan. Water tanks need to be kept above or below certain temperatures. The "preheated" range being described here is distinctly unsafe due to various microorganisms IIUC.
Post reply on HN