11–20 of 69 posts
I wonder what the environmental costs of most American homes, stores and offices using A/C at full blast 24/7/365.
Is this true in a marginal cost sense? I was under the impression most of the environmental impact occurred during the training stage, and that it was significantly less costly post training?
You could argue that this is no longer the case once the model is done; the cost per request will go down over time, as the set amount of power and coolant pumped through data centres gets divided over more people. However, AI companies can't afford to stand still. They have to keep training or they risk being made irrelevant by whatever AI company comes next. Furthermore, a non-significant amount of energy and cooli…
Really good point to put this into perspective. I tried models locally and my gpu was running red hot. Granted, I think the server boards like H100 are more optimized for the AI workloads so they run more efficiently than consumer gpus, but I don't believe they are more than 1 magnitude more efficient.
I wonder what the environmental costs of most American homes, stores and offices using A/C at full blast 24/7/365.
And whatabout all of the lights I can see being used in the average American house at night, shouldn’t we address the environmental costs of that first?
Does anyone else have a hard time accepting these calculations? I don’t doubt the serious environmental costs of AI but some of the claims in this infographic seem far-fetched. Inference costs should be much lower than training costs. And, if a 100-word email with GPT-4 requires 0.14 kWh of energy, power AI users and developers must be consuming 100x as much. Also, what about running models like Llama-3 locally? Woul…
If I run some simple inference locally on a 4090 (450 TDW card) it takes order of seconds and that sucker's going full blast, you're looking at order of 1 kJ, which is significantly higher than what is quoted in the article. Article numbers line up better with CPU inference for ~1s.
Edit: Nah I’m convinced, look at table 1. Inference costs are around 20mL in a datacenter environment.
Is this true in a marginal cost sense? I was under the impression most of the environmental impact occurred during the training stage, and that it was significantly less costly post training?
Earlier quoted context omitted.
If I run some simple inference locally on a 4090 (450 TDW card) it takes order of seconds and that sucker's going full blast, you're looking at order of 1 kJ, which is significantly higher than what is quoted in the article. Article numbers line up better with CPU inference for ~1s.
I’m still kind of skeptical. M-series Apple hardware doesn’t even get warm during inference with some local models. Edit: Nah I’m convinced, look at table 1. Inference costs are around 20mL in a datacenter environment.
Does anyone else have a hard time accepting these calculations? I don’t doubt the serious environmental costs of AI but some of the claims in this infographic seem far-fetched. Inference costs should be much lower than training costs. And, if a 100-word email with GPT-4 requires 0.14 kWh of energy, power AI users and developers must be consuming 100x as much. Also, what about running models like Llama-3 locally? Woul…
Does anyone else have a hard time accepting these calculations? I don’t doubt the serious environmental costs of AI but some of the claims in this infographic seem far-fetched. Inference costs should be much lower than training costs. And, if a 100-word email with GPT-4 requires 0.14 kWh of energy, power AI users and developers must be consuming 100x as much. Also, what about running models like Llama-3 locally? Woul…
(I can't access the article.)
Does anyone else have a hard time accepting these calculations? I don’t doubt the serious environmental costs of AI but some of the claims in this infographic seem far-fetched. Inference costs should be much lower than training costs. And, if a 100-word email with GPT-4 requires 0.14 kWh of energy, power AI users and developers must be consuming 100x as much. Also, what about running models like Llama-3 locally? Woul…
Even if the costs were lower, the trend is towards more inference compute time (o1), so these costs might be valid for the future.