[flagged]
Performance per dollar is getting faster and cheaper
111–120 of 151 posts
Re: Performance per dollar is getting faster and cheaper
#112Earlier quoted context omitted.
The problem these days is lack of nuance. It should seem entirely reasonable to be pro-datacenters-if-they're-done-right, but it feels like there are only two sides to any issue. Gas turbine whine noise isn't coming from the data center, it's being used to power the data center, but the camp is either pro data center or not, and fuck any nuance.
The problem is people keep trying to regulate businesses by name instead of by the effects they have. If we had regulations on noise, vibration, emissions, water use, electromagnetic radiation, whatever else, then it wouldn’t matter what people tried to build — if it fits within the guidelines great, otherwise back to the drawing board. Putting “data center” in your ordinances is as lazy and ineffective as putting “a…
> If we had regulations on noise, vibration, emissions, water use, electromagnetic radiation, whatever else, then it wouldn’t matter what people tried to build
We certainly do! It’s just often overridden and ignored for these companies and data centers
Re: Performance per dollar is getting faster and cheaper
#113*especially as many currencies weaken
Re: Performance per dollar is getting faster and cheaper
#114Earlier quoted context omitted.
A DGX B200 costs like ~$0.5 M and uses around 14 kW. If you plan to run it straight for 8 years 100% max usage thats around 1 GWhr. A gigawatt hour is a lot of energy but its not that much compared to the price of the actual machine. In Germany for example with its expensive energy thats about €100k worth, which spread over 8 years is pretty minor compared to the up front half mill. The real issue with high power con…
It’s more than power supply. Cooling and ventilation becomes a MUCH bigger deal at rack scale, and that costs electricity too.
With liquid cooling technologies (direct or rear door heat exchange), cooling efficiently is easier when compared to a decade ago, and it's pretty efficient when you compare the power consumption numbers (server total vs. cooling total).
See PUE (Power Usage Effectiveness) for its scientific form.
Re: Performance per dollar is getting faster and cheaper
#115That sounds literally impossible.
Agreed. The writer is pretty loose with their comparisons:
* What does it mean for "performance per dollar" to get faster? Higher, maybe; rise faster than it has in the past, maybe, but just "faster"? Nope.
* The article cites some equipment as being "2x cheaper". I think they mean "half the cost", but if so they should say it.
Re: Performance per dollar is getting faster and cheaper
#116Can you folks add performance per watt as a metric to these comparisons, I honestly want to understand where AMD fits in the stack in terms of actual performance to dollars. I have had talks with companies wanting to build data centers outside of US and find it hard to source anything Nvidia in sufficient capacity and scale. If AMD is competitive performance per watt and roughly reliable in terms of software support…
A DGX B200 costs like ~$0.5 M and uses around 14 kW. If you plan to run it straight for 8 years 100% max usage thats around 1 GWhr. A gigawatt hour is a lot of energy but its not that much compared to the price of the actual machine. In Germany for example with its expensive energy thats about €100k worth, which spread over 8 years is pretty minor compared to the up front half mill. The real issue with high power con…
> but the limited powersupply you can get for a datacenter.
Since many people haven't seen 10MW cabling for a data center or how a big GPU server is cabled, they naturally imagine connecting servers is akin to plugging an appliance to a wall.
When the electricity provider says "I neither have the capacity, nor the required cables in that area", thing gets real.
Re: Performance per dollar is getting faster and cheaper
#117Earlier quoted context omitted.
Personally, I can't wait till something like this starts getting to consumer level. https://www.anuragk.com/blog/posts/Taalas.html
That’s pretty fascinating, Apple has some innocuous LLMs and transformers baked into its devices and leveraging their neural chipset So I could see something like this where the neural chipset has an LLM that cant be so easily updated baked into it, until you get a new device
Exactly, it'd be the same as regular chip designed evolving. You get a specific model version baked into the chip, if it does what you need then it's fine. If you need more capability in the future, you just buy a new chip.
I also think the dynamic would be really different if model inference can run at ridiculous speeds. You could make a genetic algorithm loop around it, so it can generate a population of proposals at each step, then have those tested and whittled down iteratively. If inference happens at thousands of tokens per second, then from user perspective it would still be really fast, and even a small model could solve complex problems.
Re: Performance per dollar is getting faster and cheaper
#118Earlier quoted context omitted.
The people having glass literally break from the vibrations would probably disagree with your opinion https://youtu.be/_bP80DEAbuo?is=sg09k66iutKFIFSo Yet here we are, discussing "data center" as if they're standardized and of similar (nose) isolation. There are no meaningful regulations in building them, and they can be incredibly polluting. So your experience with a potentially well isolated one is sadly not the no…
The fact that people have lived and worked near data centres for decades and didn't even know what the term meant - let alone be adversely impacted by them - probably indicates they're broadly an non issue. All of a sudden out of nowhere, AI and data centres got intermingled by the media and now people seem to have big issues with them.
Because the dynamics have shifted enormously inside the rack.
10 years ago, I was running 4 CPU servers with 48 cores and 128GB of RAM in 2U enclosures with a maximum power consumption of 500W or so. I was able to stick ~20 of them in a 42U rack, totaling 10kW.
A data center full of these can be cooled with CRACs and hot/cold aisles without much problem. This is still too much for a bog-standard server colocation operation, but for HPC, that was normal and manageable.
Now, a ~1U server houses 4 SOTA NVIDIA GPUs, 64 cores, magnitudes more RAM. This server alone uses ~3KW of power. This means you go anywhere between 30kW to 50kW per rack, and you have many racks.
Of course this means more power comes in, more heat comes out. This means more sophisticated infrastructure: bigger and beefier primary and secondary power systems, beefier cooling, more heat, more noise, in short "more of everything".
Of course when you cram this much energy and heat into a relatively small space, its effect on the environment will be much more pronounced.
Facebook's previous SOTA datacenter used water infused, HEPA filtered free flowing air accross the datacenter. Now, it's server level direct liquid cooling with extensive water treatment and oversight on coolant parameters.
Compare this having a hand warmer vs. coal ember in your hand. The latter needs a much more elaborate setup to prevent it burning you badly.
Re: Performance per dollar is getting faster and cheaper
#119While cool, quantization to FP4 is practically never lossless in actual use. A lot of providers are advertising high TPS on Kimi and GLM, but the models are functionally lobotomized and no longer close to frontier quality. Would love to see this not be true.
MI355X can perform FP6 operations with the same speed as their FP4 (unique to AMD) - people should be making MXFP6 quants which would be pretty much lossless, and much closer to FP4 performance than FP8
That can only be true if the workload is compute bound, not memory bandwidth bound.
Re: Performance per dollar is getting faster and cheaper
#120Can you actually rent an MI355X per hour anywhere right now?