21–30 of 69 posts
Earlier quoted context omitted.
I’m still kind of skeptical. M-series Apple hardware doesn’t even get warm during inference with some local models. Edit: Nah I’m convinced, look at table 1. Inference costs are around 20mL in a datacenter environment.
Datacenters aren't running M-series chips.
If the machine can make that happen in a second, and it takes me 600x longer to do that while consuming power that entire time, I suspect that the AI carries an overall advantage when you actually include the full story.
I'll file this under "lazy tech journalists didn't bother to do the math past the first, most obvious step".
I think the comparison that should be made here is the actual alternative - what is the cost in kw/hr of me sitting in front of my powered-on and not-sleeping laptop while I spend the next 10 minutes penning an email? If the machine can make that happen in a second, and it takes me 600x longer to do that while consuming power that entire time, I suspect that the AI carries an overall advantage when you actually inclu…
I think the comparison that should be made here is the actual alternative - what is the cost in kw/hr of me sitting in front of my powered-on and not-sleeping laptop while I spend the next 10 minutes penning an email? If the machine can make that happen in a second, and it takes me 600x longer to do that while consuming power that entire time, I suspect that the AI carries an overall advantage when you actually inclu…
I think the comparison that should be made here is the actual alternative - what is the cost in kw/hr of me sitting in front of my powered-on and not-sleeping laptop while I spend the next 10 minutes penning an email? If the machine can make that happen in a second, and it takes me 600x longer to do that while consuming power that entire time, I suspect that the AI carries an overall advantage when you actually inclu…
Are you gonna get less done in those 8 hours? Probably. Is it good for the environment to get more done in less time? Can't really answer that one generically. We humans do a lot of stuff simply because we can.
If unemployment goes up 20% from generative AI (just a made up number for narrative purposes), are those 20% not gonna sit in front of a screen all day? Are they perhaps even going to play games that max out their machines instead?
Of course another way to see all this is to argue that we need 20% less humans now, so we save any environmental costs those people induce. Seems a bit gruesome to me.
I think the comparison that should be made here is the actual alternative - what is the cost in kw/hr of me sitting in front of my powered-on and not-sleeping laptop while I spend the next 10 minutes penning an email? If the machine can make that happen in a second, and it takes me 600x longer to do that while consuming power that entire time, I suspect that the AI carries an overall advantage when you actually inclu…
Of course, if it's streamlined to be 600x faster, it's just going to be done 600x more often.
Is this true in a marginal cost sense? I was under the impression most of the environmental impact occurred during the training stage, and that it was significantly less costly post training?
Llama 403b takes OOM a kilowatt minute to respond on our local gpu server, or about 10 grams of C02 per email. Last I checked, add another 20 grams of amortized manufacturing emissions. A typical commute is OOM 5-10 kg of CO2. this article is alarmist bullshit. (for entirely unrelated reasons openai delenda est)
That sounds pretty bad still, no?
If the goal is to reduce overall usage, what to stop should be determined by value, not chronologically/LIFO.