181–190 of 273 posts
"TensorWave is a cloud provider specializing in AI workloads. Their platform leverages AMD’s Instinct™ MI300X accelerators, designed to deliver high performance for generative AI workloads and HPC applications." I suggest taking the report with a grain of salt.
Well, there's the beauty of specifying exactly how you ran your benchmark, it is easy to reproduce and disprove or confirm (assuming you got the hardware).
Earlier quoted context omitted.
The salt is in the plain sight. The do the standard AMD comparison: 8x AMD MI300X (192GB, 750W) GPU 8x H100 SXM5 (80GB, 700W) GPU The fair comparison would be against 8x H100 NVL (188GB, Price tells a story. If AMD performance would be in par with Nvidia they would not sell their cards for 1/4 price.
MTr ------------------ H100 SXM5 80,000 MI300X 153,000 H100 NVL 160,000 H100 SXM4 has 52% of the transistors MI300X has, half of the RAM and MI300X achieves *ONLY* 33% higher throughput compared to the H100. MI300X was launched 6 months ago, H100 20 months ago. AMD has work to do.
Earlier quoted context omitted.
N26, Klarna and Revolut are not tech companies. Neobanks are still banks. And Klarna is just modern store credit cards. These have been around for decades. Adyen is very underrated, and Spotify is definitely tech. Stripe should be on the list. DeepMind at one point.
Stripe was founded in California. It's an American business that focused exclusively on the American domestic market in their first years of operation. Many tech companies in the US are founded by immigrants from Europe and elsewhere. That the Collisons chose to start their business in the States is no coincidence.
It's just much cheaper and easier for start-ups if you're developing a SW product to sell it in the US market first and only when you've made money there, slowly bring it in the EU.
Starting off SW products in the EU is suicide (unless you're targeting some niche in the local market that's safe from competitors from abroad because it ties into some local idiosyncrasies on language, culture and law).
Earlier quoted context omitted.
Thx! Anyone who says Nivida isnt king, needs a reality check.
I love AMD, but my Nvidia stock position currently is much higher than AMD.
Earlier quoted context omitted.
MTr ------------------ H100 SXM5 80,000 MI300X 153,000 H100 NVL 160,000 H100 SXM4 has 52% of the transistors MI300X has, half of the RAM and MI300X achieves *ONLY* 33% higher throughput compared to the H100. MI300X was launched 6 months ago, H100 20 months ago. AMD has work to do.
Maybe I'm a naive fanboy, but I would put my money on Apple catching Nvidia before AMD or Intel.
Earlier quoted context omitted.
where can I rent a H100 for 4-5 dollars an hour? AWS doesn't let you use p5 instances (not getting a quota as a private person), lambda cloud is sold out.
It looks like Runpod currently (checked right now) has "Low" availability of 8x MI300 SXM (8x$4.89/h), H100 NVL (8x$4.39/h), and H100 (8x$4.69/h) nodes for anyone w/ some time to kill that wants to give the shootout a try.
Earlier quoted context omitted.
some people emit llvm ir (maaaaybe ptx) directly instead of using the C/C++ frontend to CUDA. that's absolutely the only optional part of the stack and also basically the most trivial (i.e., it's not the frontend that's hard but the target codegen).
LLVM IR to machine code is not the part that AMD has traditionally struggled with. What you call "trivial" is. If everyone started emitting IR and didn't rely on NVidia-owned libs then the space would become unrecognizable. The codegen is something AMD has always been decent at, hence them beating NVidia in compute benchmarks for most of the past 20 years.
alright fine it's the codegen and the runtime and the driver and the library ecosystem...
> If everyone started emitting IR and didn't rely on NVidia-owned libs then the space would become unrecognizable.
I have no clue what this means - which libs are you talking about here? the libs that contain the implementations of their runtime? or the libs that contain the user space components of their driver? or the libs that contain their driver and firmware code? And exactly which of these will "everyone emitting IR" save us from?
They also used Flywheel for AMD while not bothering to turn on Flywheel for Nvidia, which is crazy since Flywheel improves Nvidia performance by 70%. https://mk1.ai/blog/flywheel-launch
In this context the 33% performance lead by AMD looks terrible, and straight up looks slower.
Earlier quoted context omitted.
For understanding: What would be a suitable input length in your oppinion? And why isnt this a good one: Are real-life queries shorter? Or longer? If i count one word as a token, then in my case most of the queries are less than 128 words.
IMO the relevant benchmark for now is a mixed stream of requests with 50 (20%), 500 (50%), 2000 (10%) and 50k (20%) input tokens, ignore EOS and decode until you get around 300 output tokens.
I tried to look for some service provider to publish this kind of metrics, but haven't found any.
Also curious how many companies are dropping that much money on those kind of accelerators just to run 8x 7B param models in parallel... You're also talking about being able to train a 14B model on a single accelerator. I'd be curious to see how "full-accelerator train and inferrence" workloads would look ie: Training a 14B param model then inferrence throughput on a 4x14B workload.
AMD (and almost every other inferrence claim maker so far... intel and apple specifically) have consistently cherry picked the benchmarks to claim a win over, and ignored the remainder which all show nvidia in the lead - and they've used mid-gen comparison models as many commenters here pointed out in this article.