Performance per dollar is getting faster and cheaper • Hacker News Board

Re: Performance per dollar is getting faster and cheaper

#61

Post by psychoslave » Sat, Jul 04, 2026, 6:11 AM UTC

dannyw wrote: post #48

kingstnap wrote: post #21

Earlier quoted context omitted.

A DGX B200 costs like ~$0.5 M and uses around 14 kW. If you plan to run it straight for 8 years 100% max usage thats around 1 GWhr. A gigawatt hour is a lot of energy but its not that much compared to the price of the actual machine. In Germany for example with its expensive energy thats about €100k worth, which spread over 8 years is pretty minor compared to the up front half mill. The real issue with high power con…

It’s more than power supply. Cooling and ventilation becomes a MUCH bigger deal at rack scale, and that costs electricity too.

And datacenters have impact on everything around them. If at the end of the day to result is a few more yachts and jets and, a lot more of miserable humans starving in ruined ecosystems, maybe that’s not the best go-to direction.

Re: Performance per dollar is getting faster and cheaper

#62

Post by unrvl22 » Sat, Jul 04, 2026, 6:17 AM UTC

hassaanr wrote: post #41

While cool, quantization to FP4 is practically never lossless in actual use. A lot of providers are advertising high TPS on Kimi and GLM, but the models are functionally lobotomized and no longer close to frontier quality. Would love to see this not be true.

MI355X can perform FP6 operations with the same speed as their FP4 (unique to AMD) - people should be making MXFP6 quants which would be pretty much lossless, and much closer to FP4 performance than FP8

Re: Performance per dollar is getting faster and cheaper

#63

Post by unrvl22 » Sat, Jul 04, 2026, 6:22 AM UTC

technoabsurdist wrote: post #10

AussieWog93 wrote: post #6

The 2600 tok/s is an "aggregate", not the actual throughput.

yes it is 213 tok/s single stream (so per user)

that 213 wasn't achieved when saturated though. was probably more like 30 tps per stream when doing 2.6k tps.

Re: Performance per dollar is getting faster and cheaper

#64

Post by unrvl22 » Sat, Jul 04, 2026, 6:26 AM UTC

nullc wrote: post #22

Schiendelman wrote: post #18

I'm not surprised to see competition with Blackwell. Rubin is 5x faster than Blackwell at inference - Blackwell is the last generation Nvidia didn't optimize specifically for inference. If I'm missing something, please let me know!

how do you get 5x faster at inference when inference is memory bandwidth limited? getting 5x the memory bandwidth of a h100 seems physically difficult.

inference is only memory bandwidth limited when targeting higher tps / high single stream tps. the weights only need to be moved across once per forward pass, when you batch say 100 streams per forward pass (which is what most inference services do / care about) its compute bottlenecked.

Re: Performance per dollar is getting faster and cheaper

#65

Post by butvacuum » Sat, Jul 04, 2026, 6:37 AM UTC

psychoslave wrote: post #61

dannyw wrote: post #48

Earlier quoted context omitted.

It’s more than power supply. Cooling and ventilation becomes a MUCH bigger deal at rack scale, and that costs electricity too.

And datacenters have impact on everything around them. If at the end of the day to result is a few more yachts and jets and, a lot more of miserable humans starving in ruined ecosystems, maybe that’s not the best go-to direction.

You say they have a large impact, but having lived somewhere with some of the largest data centers- they very much don't. At least not more so then any other structure that paves over greenery.

love to debate actual discission points. pull up "datacenter dfw" on google maps for mine.

Re: Performance per dollar is getting faster and cheaper

#66

Post by zozbot234 » Sat, Jul 04, 2026, 6:37 AM UTC

hassaanr wrote: post #41

While cool, quantization to FP4 is practically never lossless in actual use. A lot of providers are advertising high TPS on Kimi and GLM, but the models are functionally lobotomized and no longer close to frontier quality. Would love to see this not be true.

Kimi uses INT4 as its native format, there's no such thing as "better than 4-bit precision" for that model. This is in contrast with GLM for which 16-bit precision is native and 8-bit is in common use.

Re: Performance per dollar is getting faster and cheaper

#67

Post by ffsm8 » Sat, Jul 04, 2026, 6:55 AM UTC

butvacuum wrote: post #65

psychoslave wrote: post #61

Earlier quoted context omitted.

And datacenters have impact on everything around them. If at the end of the day to result is a few more yachts and jets and, a lot more of miserable humans starving in ruined ecosystems, maybe that’s not the best go-to direction.

You say they have a large impact, but having lived somewhere with some of the largest data centers- they very much don't. At least not more so then any other structure that paves over greenery. love to debate actual discission points. pull up "datacenter dfw" on google maps for mine.

The people having glass literally break from the vibrations would probably disagree with your opinion

https://youtu.be/_bP80DEAbuo?is=sg09k66iutKFIFSo

Yet here we are, discussing "data center" as if they're standardized and of similar (nose) isolation.

There are no meaningful regulations in building them, and they can be incredibly polluting. So your experience with a potentially well isolated one is sadly not the norm going forward. And we don't even know how close you lived, if you're eg talking about "within 5km/3miles" then your experience would also have little value in this discussion in general.

Re: Performance per dollar is getting faster and cheaper

#68

Post by calin2k » Sat, Jul 04, 2026, 7:20 AM UTC

then why is token per dollar getting more expensive?

Re: Performance per dollar is getting faster and cheaper

#69

Post by AtlasBarfed » Sat, Jul 04, 2026, 7:21 AM UTC

calin2k wrote: post #68

then why is token per dollar getting more expensive?

Because they are dumping/subsidizing it token processing to try and get companies to fire as many people as possible. So they'll be dependent upon the companies when they have to Jack the rates

Re: Performance per dollar is getting faster and cheaper

#70

Post by jml7c5 » Sat, Jul 04, 2026, 7:24 AM UTC

ffsm8 wrote: post #67

butvacuum wrote: post #65

Earlier quoted context omitted.

You say they have a large impact, but having lived somewhere with some of the largest data centers- they very much don't. At least not more so then any other structure that paves over greenery. love to debate actual discission points. pull up "datacenter dfw" on google maps for mine.

The people having glass literally break from the vibrations would probably disagree with your opinion https://youtu.be/_bP80DEAbuo?is=sg09k66iutKFIFSo Yet here we are, discussing "data center" as if they're standardized and of similar (nose) isolation. There are no meaningful regulations in building them, and they can be incredibly polluting. So your experience with a potentially well isolated one is sadly not the no…

>The people having glass literally break from the vibrations would probably disagree with your opinion

Can you cite a source for this? It's not in the video, as far as I can tell.

I would be wary of Benn Jordan's videos. They are full of mistakes and misrepresentations, as Andy Masley has convincingly demonstrated: https://blog.andymasley.com/p/contra-benn-jordan-data-center...

I recall seeing Benn Jordan's responses on Bluesky and thinking they were quite poor. He was unwilling to admit to mistakes, and kept trying to grasp at newly searched papers that didn't actually support his arguments.