While cool, quantization to FP4 is practically never lossless in actual use. A lot of providers are advertising high TPS on Kimi and GLM, but the models are functionally lobotomized and no longer close to frontier quality. Would love to see this not be true.
Performance per dollar is getting faster and cheaper
41–50 of 151 posts
Re: Performance per dollar is getting faster and cheaper
#42This is very interesting and yet not at the same time. This looks to be optimized for single-stream LLM traffic which is not viable to serve in a production setting. It's only interesting to hobbyists that want to run the model locally. It's genuinely neat that AI can find the right optimization pathways in an AMD inference server to unlock this but at the same token (pun-intended) this is a classic case of benchmark…
hi yes it’s not optimized for single stream it’s optimized for total node throughput
Oh, that's much better then. A good metric to share is the tokens per second per user for the node rather than the total throughput of the node. It disambiguates what's being optimized for much better than your blog post currently does.
Re: Performance per dollar is getting faster and cheaper
#43This is very interesting and yet not at the same time. This looks to be optimized for single-stream LLM traffic which is not viable to serve in a production setting. It's only interesting to hobbyists that want to run the model locally. It's genuinely neat that AI can find the right optimization pathways in an AMD inference server to unlock this but at the same token (pun-intended) this is a classic case of benchmark…
You got it backwards; it's ~200 on single stream so the 2,600 is achieved with ~13 streams.
Yeah that makes sense. I'm more familiar with seeing tok/s/user + TTFT rather than the total node throughput.
Re: Performance per dollar is getting faster and cheaper
#44There’s noticeable accuracy degradation when they switched from fp8 to mxfp4
Wafer discontinued their own "Wafer Pass" flagship coding plan within weeks of launch and had to issue prorated refunds. Now they're bragging about squeezing costs down even further via quantization, even though their implementation is clearly lacking.
[1] https://www.ycombinator.com/launches/Q9i-wafer-pass-flat-rat...
Re: Performance per dollar is getting faster and cheaper
#45Earlier quoted context omitted.
You're right - Rubin is better at NVFP4 training , not inference, thank you for catching me!
What does it mean it's better at nvfp4 training? What's different between training and inference to make this true?
I'm also puzzled by that statement. The issue with training is (as I understand it) one of precision and the associated numerical stability. You need enough bits in order for backprop to function correctly.
Of course there are techniques such as quantization aware training but I don't understand why a datatype would work for inference but not for that.
You can also abandon backprop entirely but that comes with a whole host of tradeoffs and again why would it work for inference but not for whatever alternative training regime you selected?
Re: Performance per dollar is getting faster and cheaper
#46Earlier quoted context omitted.
You're right - Rubin is better at NVFP4 training , not inference, thank you for catching me!
What does it mean it's better at nvfp4 training? What's different between training and inference to make this true?
We're getting to the limit of my understanding, but I believe most Blackwell users still usually run FP8 passes through the transformer engine - they'll just store weights at NVFP4. Nvidia has model-specific stabilization recipes for NVFP4 end to end, but they're taking fixes all the time.
Nvidia says Rubin should have fewer stability problems training with FP4 because of hardware changes - "adaptive compression". There will still be outlier instability inherently, but something they're designing in reduces the cost of managing it.
But yeah, grain of salt - we haven't seen this in practice.
Re: Performance per dollar is getting faster and cheaper
#47Agentic coding drivers for different architectures is a massive unlock for the world So much compute is under utilized waiting for a savant or company to prioritize an architecture, and now all the other engineers can tackle this at any time if they get inspired on the right prompts
this is exactly our thesis at wafer :) thank you for the support
well done
Re: Performance per dollar is getting faster and cheaper
#48Can you folks add performance per watt as a metric to these comparisons, I honestly want to understand where AMD fits in the stack in terms of actual performance to dollars. I have had talks with companies wanting to build data centers outside of US and find it hard to source anything Nvidia in sufficient capacity and scale. If AMD is competitive performance per watt and roughly reliable in terms of software support…
A DGX B200 costs like ~$0.5 M and uses around 14 kW. If you plan to run it straight for 8 years 100% max usage thats around 1 GWhr. A gigawatt hour is a lot of energy but its not that much compared to the price of the actual machine. In Germany for example with its expensive energy thats about €100k worth, which spread over 8 years is pretty minor compared to the up front half mill. The real issue with high power con…
It’s more than power supply. Cooling and ventilation becomes a MUCH bigger deal at rack scale, and that costs electricity too.
Re: Performance per dollar is getting faster and cheaper
#49While cool, quantization to FP4 is practically never lossless in actual use. A lot of providers are advertising high TPS on Kimi and GLM, but the models are functionally lobotomized and no longer close to frontier quality. Would love to see this not be true.
First thing I noticed as well
Re: Performance per dollar is getting faster and cheaper
#50Agentic coding drivers for different architectures is a massive unlock for the world So much compute is under utilized waiting for a savant or company to prioritize an architecture, and now all the other engineers can tackle this at any time if they get inspired on the right prompts
[dead]