181–190 of 221 posts
Earlier quoted context omitted.
Wow, are these 7 RTX 4090s in a single setup? Care to share more how you build it (case, cooling, power, ..)?
Most of these are just an EPYC server platform, some cursed risers and multiple PSUs (though cryptominer server PSU adapters are probably better). See https://nonint.com/2022/05/30/my-deep-learning-rig/ and https://www.mov-axbx.com/wopr/wopr_concept.html .
Earlier quoted context omitted.
The problem is that the specs of AMD consumer-grade GPUs do not translate to computer performance when you try and chain more than one together. I have 7 NVidia 4090s under my desk happily chugging along on week long training runs. I once managed to get a Radeon VII to run for six hours without shitting itself.
Wow, are these 7 RTX 4090s in a single setup? Care to share more how you build it (case, cooling, power, ..)?
Beyond that, not much else.
Earlier quoted context omitted.
Reality check for anyone considering this: I just got a used 3090 for $900 last month. It works great. I would not recommend buying one for $600, it probably either won’t arrive or will be broken. Someone will reply saying they got one for $600 and it works, that doesn’t mean it will happen if you do it. I’d say the market is realistically $900-1100, maybe $800 if you know the person or can watch the card running fir…
I've bought 5 used and they're all perfect. But that's what buyer protection on ebay is for. Had to send back an Epyc mobo with bent pins and ebay handled it fine.
Just beware, the card might be "working fine" on a first glance, but actually be damaged.
Great, I have yet to understand why does not the ML community really push or move away from CUDA? To me, it feel like a dinosaur move to build on top of CUDA which is screaming proprietary nothing about it is open source or cross platform. The reason why I say its dinosaur is, imagine, we as a dev community continued to build on top of Flash or Microsoft Silverlight... LLM and ML has been out for quiet a while, with…
For me personally, hacking together projects as a hobbiest, 2 reasons : 1. It just works. When i tried to build things on Intel Arcs, i spent way more hours bikeshedding ipex and driver issues than developing 2. LLMs seem to have more cuda code in their training data. I can leverage claude and 4o to help me build things with cuda, but trying to get them to help me do the same things on ipex just doesn't work. I'd ver…
Earlier quoted context omitted.
Well sure, but in other GPU tasks, like Raytracing, the difference between these GPUs is far more pronounced. And AMD has passable Raytracing units (NVidias are better but the difference is bigger than these LLM results). If RAM is the main bottleneck then CPUs should be on the table.
> If RAM is the main bottleneck then CPUs should be on the table That's certainly not the case. The graphics memory model is very different from the CPU memory model. Graphics memory is explicitly designed for multiple simultaneous reads (spread across several different buses) at the cost of generality (only portions of memory may be available on each bus) and speed (the extra complexity means reads are slower). This…
I have been playing around with Phi-4 Q6 on my 7950x and 7900XT (with HSA_OVERRIDE_GFX_VERSION). It's bloody fast, even with CPU alone - in practical terms it beats hosted models due to the roundtrip time. Obviously perf is more important if you're hosting this stuff, but we've definitely reached AMD usability at home.
Earlier quoted context omitted.
The Radeon VII is special compared to most older (and current) affordable GPUs in that it used HBM giving it memory bandwidth comparable to modern cards ~1TB/s and has reasonable FP64 (1:4) throughput instead of (1:64). So this card can still be pretty interesting for running memory bandwidth intensive FP64 workloads. Anything affordable afterward by either AMD or Nvidia crippled realistic FP64 throughput to below wh…
If we speak about FP64, are your loads more like fluid dynamics than ML training?
Earlier quoted context omitted.
Interesting. I wonder if focusing on GPUs and CPUs is something that requires two companies instead of one, whether the concentration of resources just leads to one arm of your company being much better than the other.
Nvidia maintains a competitive cpu...
Earlier quoted context omitted.
On the other hand, for double precision a Radeon Pro VII is many times faster than a RTX 4090 (due to 1:2 vs. 1:64 FP64:FP32 ratio). Moreover, for workloads limited by the memory bandwidth, a Radeon Pro VII and a RTX 4090 will have about the same speed, regardless what kind of computations are performed. It is said that speed limitation by memory bandwidth happens frequently for ML/AI inferencing.
Double precision is not used in either inference or training as far as I know.
Because the previous poster had mentioned only single precision, where RTX 4090 is better, I had to complete the data with double precision, where RTX 4090 is worse, and memory bandwidth where RTX 4090 is the same, otherwise people may believe that progress in GPUs over 5 years has been much greater than it really is.
Moreover, memory bandwidth is very relevant for inference, much more relevant than FP32 throughput.