Earlier quoted context omitted.
What workloads do you run that require fp64?
Simulation.
Radeon VII is the best bang for the bucks in the FP64 space, then older K80s. Titan V/V100 are still expensive.
Benchmarking TensorFlow on Nvidia GeForce RTX 3090
91–100 of 104 posts
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#92Earlier quoted context omitted.
Power doesn't cost nearly as much for TCO to get anywhere near even the "preemptible price" of V100 (and probably A100 when it's ready) over a period of half a year. And now that 3090 has 24gb, which is needed for larger models, a solution with a couple of consumer cards is even more competitive for experimentation. You can also sell your cards in a year or two, and recover some of the costs. (All that of course if N…
So for the people who I know do it professionally, the answer is really simple - when they run calculations for clients, they can add GCP compute time on the invoice. You can't bill a client for your own electricity usage. I mean you can factor it into the price of your service, but then that brings a whole pile of other issues with it.
Cloud if:
- Someone else is paying
- You expect to dabble
- You need burst capability
Buy if:
- Cost sensitive & capable
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#93This is a poor comparison of performance. All of these networks are CNNs, and very old architectures at that. They are all probably memory bottlenecked which is why you see the consistent 50% improvement in FP32 perf. It is also not clear what batch sizes are being used for any of the tests. If you switch to FP16 training, you must increase the batch size to properly utilize the Tensor Cores. If you compare these car…
For many of us, the Inception-style CNN workloads--especially at FP32--are much more realistic than large language models that may be better suited to take advantage of the tensor cores. If I'm going to be memory bottlenecked either way, I probably don't want to spend an extra $1000 on 400 tensor cores I can't take full advantage of.
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#94Earlier quoted context omitted.
just make sure it's NVidia. whatever graphics card you want -- all their consumer cards will work great for deep learning. make sure your motherboard and processor support whatever the newest version of PCIe is -- a major factor with deep learning is bandwidth moving data on/off the GPU. AMD GPUs can theoretically be used for machine learning, but right now software support is lacking -- you will spent more time conf…
This is also a reason why it might make sense to hold off unless you have some kind of time-sensitive project. Nvidia came to dominate the market at a time when AMD wasn't making particularly competitive GPUs, but that isn't really the case anymore. For anything not so expensive that nobody is really going to buy it anyway, the current and expected (in less than a month) AMD GPUs are competitive on performance. The r…
I tried to go red twice. Red team has been winning at perf/$ for a decade! I thought I did my homework and established compatibility and suitability for the purposes I cared about. Unfortunately, both times I eventually ran into unanticipated incompatibilities I couldn't work around. I wound up paying the green tax anyway and also the price spread + ebay fees. Oof.
Twice bitten... once shy? In any case, I'm going to let someone else be the guinea pig this time.
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#95Earlier quoted context omitted.
Here's hoping for an A100 titan with un-nerfed FP64. The 3090 is twice as nerfed as previous generations, which were also bad at 1:32. Now it's 1:64 :(
It seems that Radeon VII and Titan V are the last cards with decent fp64 performance for foreseeable future. Both Nvidia and AMD now basically have different architectures for their consumer and data-center products.
Yep, sure looks that way. I'll still be dreaming of a Titan A!
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#96Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#97Earlier quoted context omitted.
This is also a reason why it might make sense to hold off unless you have some kind of time-sensitive project. Nvidia came to dominate the market at a time when AMD wasn't making particularly competitive GPUs, but that isn't really the case anymore. For anything not so expensive that nobody is really going to buy it anyway, the current and expected (in less than a month) AMD GPUs are competitive on performance. The r…
i think most people would just use TF/PyTorch and ignore the specific technology on the backend. not much GPU specific stuff to learn -- very, very few deep learning people write their own CUDA code. so the question is just -- when will it be very simple to install these packages for AMD GPUs, with enough mathematical operations implemented and optimized to let you do the things you want to do. right now things sort…
> very few deep learning people write their own CUDA code.
True, but even once TF/PyTorch support AMD well it's highly possible that an unanticipated CUDA dependency will pop up in one's computational journey. NVidia subsidized CUDA seminars for a decade and now it's all over the place, both in the flagship frameworks and in the nooks and crannies.
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#98Earlier quoted context omitted.
There will be 20GB version of 3080 soon.
what would be the benefit there? 20GB can't be that much cheaper than 24GB, right?
It will be for the gamers who think that 10 GB isn't enough VRAM, and as a way for nVidia to have an answer for rumors that AMD's next GPU will have 16 GB.
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#99This is a poor comparison of performance. All of these networks are CNNs, and very old architectures at that. They are all probably memory bottlenecked which is why you see the consistent 50% improvement in FP32 perf. It is also not clear what batch sizes are being used for any of the tests. If you switch to FP16 training, you must increase the batch size to properly utilize the Tensor Cores. If you compare these car…
For many of us, the Inception-style CNN workloads--especially at FP32--are much more realistic than large language models that may be better suited to take advantage of the tensor cores. If I'm going to be memory bottlenecked either way, I probably don't want to spend an extra $1000 on 400 tensor cores I can't take full advantage of.
If I may ask, why are the Inception style workloads still popular, rather than architectures like EfficientNet?
Also, why FP32? CNNs are some of the most robust models to train in FP16 (much easier than language models) so you could get yourself a quick XXX speedup and 2x memory savings by switching over.
(btw not intending to be accusatory or anything, I just think FP16 training deserves a lot more adoption that it currently seems to have :)
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#100Earlier quoted context omitted.
Totally agree! I think 3090 could be a lot more cost effective for researchers to dabble with NLP. But it really grinds my gears when people post these misleading benchmarks... the 3090 is handicapped at half-rate tensor core performance while the Titan RTX is not. So if you're someone who does their work mainly in FP32, you will see improved performance with the 3090. On the other hand, if you are an FP16 speed demo…
What do you think about TF32 in 3090? Could it replace FP32 with 5x speedup?
I've done a lot of work in ML numerics, and I think TF32 is a completely safe drop-in for FP32 for ML workloads. NVIDIA seems to think so too, which is why on A100 it won't even be an option, it will be the default mode for any FP32 matrix multiplies.
But on 3090, I don't think the speedup will be 5x, it should be closer to like 2x. The 3090 has 35.6 TF/s at TF32 and the Titan RTX has 16.3 TF/s at FP32. Once again I think there is handicapping going on for 3090.