Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A bottle of water per email: the hidden environmental costs of using AI chatbots

washingtonpost.com

Post reply on HN
Print view View original

31–40 of 69 posts

Re: A bottle of water per email: the hidden environmental costs of using AI chatbots

#31

Post by gcr »

archerx wrote: post #3

I wonder what the environmental costs of most American homes, stores and offices using A/C at full blast 24/7/365.

That’s the tricky thing about conservation. These problems have multiple sources!

Every time a completely new energy-hungry product category is introduced, the carbon budget gets harder and harder to balance. In the 19X0s it was television and air conditioning, in the 2010s it was Bitcoin maybe, and now we’re adding AI. In the 2X00s, teleportation will double the typical commute energy usage, until heavy regulation and scientific advances will bring it down again.

Re: A bottle of water per email: the hidden environmental costs of using AI chatbots

#32

Post by marginalia_nu »

gcr wrote: post #15
marginalia_nu wrote: post #9

Earlier quoted context omitted.

If I run some simple inference locally on a 4090 (450 TDW card) it takes order of seconds and that sucker's going full blast, you're looking at order of 1 kJ, which is significantly higher than what is quoted in the article. Article numbers line up better with CPU inference for ~1s.

I’m still kind of skeptical. M-series Apple hardware doesn’t even get warm during inference with some local models. Edit: Nah I’m convinced, look at table 1. Inference costs are around 20mL in a datacenter environment.

1 kJ is for reference enough to heat 1 L (33 oz) of water by ~0.25C (~0.5F). The machine will probably heat up a few degrees if you run inference once, but since it's essentially one big heatsink it will dissipate throughout the body and into the air. The problem begins when you run it continuously, as you would in a datacenter.

Re: A bottle of water per email: the hidden environmental costs of using AI chatbots

#33

Post by johnthesecure »

ThouYS wrote: post #23

In all these calculations you have to wonder what "using water" even means. Add to that that most water cooled systems are closed loops, i.e. no water escapes

The calculations paper talks about the water cycle and explains that fresh clean water isn't well distributed. I wonder whether big evaporators like data centers might actually help to redistribute water across the planet.

Re: A bottle of water per email: the hidden environmental costs of using AI chatbots

#34

Post by gcr »

MSFT_Edging wrote: post #17
gcr wrote: post #15

Earlier quoted context omitted.

I’m still kind of skeptical. M-series Apple hardware doesn’t even get warm during inference with some local models. Edit: Nah I’m convinced, look at table 1. Inference costs are around 20mL in a datacenter environment.

Datacenters aren't running M-series chips.

Google’s TPU energy usage is a well-kept secret / competitive advantage. If energy efficiency isn’t a major concern for them, I bet it will be in a couple years.

Re: A bottle of water per email: the hidden environmental costs of using AI chatbots

#35

Post by tomrod »

llimos wrote: post #29

Just because AI is new, it doesn't mean it's less valuable than the older things using up resources. AI is far more valuable than, say, television. If the goal is to reduce overall usage, what to stop should be determined by value, not chronologically/LIFO.

Jevon's paradox is being rediscovered in a big way!

Re: A bottle of water per email: the hidden environmental costs of using AI chatbots

#36

Post by dartos »

guitarlimeo wrote: post #21
MSFT_Edging wrote: post #17

Earlier quoted context omitted.

Datacenters aren't running M-series chips.

Well not M-series chips specifically, but chips optimized for these kind of workloads (like the neural engine in M-series chips is).

IIRC The M series chip isn’t specifically optimized for ML workloads, the biggest gain it has is having unified video and cpu memory as transferring layers between the two is a big bottleneck on non Apple systems.

Real ML hardware (like the Nvidia H1000s) that can handle the kind of inference traffic you see in production get hot and use quite a bit of energy, especially when they run at full blast 24/7

Re: A bottle of water per email: the hidden environmental costs of using AI chatbots

#37

Post by yunwal »

gcr wrote: post #28
QuadmasterXLII wrote: post #4

Earlier quoted context omitted.

Llama 403b takes OOM a kilowatt minute to respond on our local gpu server, or about 10 grams of C02 per email. Last I checked, add another 20 grams of amortized manufacturing emissions. A typical commute is OOM 5-10 kg of CO2. this article is alarmist bullshit. (for entirely unrelated reasons openai delenda est)

So you can double your commute‘s environmental impact by using llama 1000x per day? That sounds pretty bad still, no?

A thousand times? I’d have a hard time typing out that many queries in 8 hours. Even 100 seems like a stretch for someone who uses it within something like cursor.

Re: A bottle of water per email: the hidden environmental costs of using AI chatbots

#38

Post by gcr »

jeroenhd wrote: post #5
Apfel wrote: post #2

Is this true in a marginal cost sense? I was under the impression most of the environmental impact occurred during the training stage, and that it was significantly less costly post training?

You could argue that this is no longer the case once the model is done; the cost per request will go down over time, as the set amount of power and coolant pumped through data centres gets divided over more people. However, AI companies can't afford to stand still. They have to keep training or they risk being made irrelevant by whatever AI company comes next. Furthermore, a non-significant amount of energy and cooli…

Another corollary is that AI companies don’t train one model at a time. Typical engineers will have maybe 5-10 models training at once. Large hyperparameter grid searches might have hundreds or thousands. Most of these will turn out to be duds. Only one model gets released, and that one’s energy efficiency is what’s reported.

Re: A bottle of water per email: the hidden environmental costs of using AI chatbots

#40

Post by Panzer04 »

marginalia_nu wrote: post #9
althea_tx wrote: post #6

Does anyone else have a hard time accepting these calculations? I don’t doubt the serious environmental costs of AI but some of the claims in this infographic seem far-fetched. Inference costs should be much lower than training costs. And, if a 100-word email with GPT-4 requires 0.14 kWh of energy, power AI users and developers must be consuming 100x as much. Also, what about running models like Llama-3 locally? Woul…

If I run some simple inference locally on a 4090 (450 TDW card) it takes order of seconds and that sucker's going full blast, you're looking at order of 1 kJ, which is significantly higher than what is quoted in the article. Article numbers line up better with CPU inference for ~1s.

1kj is nothing. That's 0.3wh, or 0.0003kwh.
Post reply on HN