See also Tim Detter's fantastic post on GPU performance (which doesn't use benchmarks for the latest cards but instead calculates performance with a model):
https://timdettmers.com/2020/09/07/which-gpu-for-deep-learni...
HN Discussion:
Benchmarking TensorFlow on Nvidia GeForce RTX 3090
71–80 of 104 posts
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#72Can someone explain the difference between fp16 and fp32 in these benchmarks because the difference is pretty dramatic. I assume it's floating point precision(?) but why would lower precision be slower relatively on the 3090? For training jobs how does the precision impact accuracy of the model? Edit: clarified that I am referring to slower relative performance
FP16 is faster (units are images per second)
3090 opted for bundling 2x FP32 units Bulldozer-style and now FP16 is processed by those cores as well, so FP16 and FP32 have the same performance (35.58TFlops).
https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#73This is a poor comparison of performance. All of these networks are CNNs, and very old architectures at that. They are all probably memory bottlenecked which is why you see the consistent 50% improvement in FP32 perf.
It is also not clear what batch sizes are being used for any of the tests. If you switch to FP16 training, you must increase the batch size to properly utilize the Tensor Cores.
If you compare these cards at FP16 performance on large language models (think GPT-style with large model dimension), I am confident you will see Titan RTX outperform the 3090. The former has 130 TF/s of FP16.32 tensor core performance while the latter has only 70 TF/s.
Link: https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/a...
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#74Earlier quoted context omitted.
If this isn't OT... Could you kindly advise what kind of computer would make sense to purchase to begin learning about ML? I was assuming I'd get a 3080. Should I get a case that could potentially house 2 x 3080's? Does the case require any special cooling considerations, or just whatever will fit the cards? What CPU would you get?
Go for it! Get a motherboard with reinforced PCIe slots for both GPUs though, the cheaper mobos only have one armoured slot. Also, you really should use the 3080 Founders Edition I guess for a dual setup as they blow out part of their heat in the back. Otherwise you need good air control and as thin 3080 cards as you can find (so there's some space between them in the case). Still waiting for good thermal benchmarks…
Woah, there. They said they were just learning. No need to purchase special hardware until you're trying to run state of the art models.
You can get very far on any laptop before hardware becomes the main blocker. And before building an ML machine, there are cloud compute options available for far cheaper.
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#75This is a poor comparison of performance. All of these networks are CNNs, and very old architectures at that. They are all probably memory bottlenecked which is why you see the consistent 50% improvement in FP32 perf. It is also not clear what batch sizes are being used for any of the tests. If you switch to FP16 training, you must increase the batch size to properly utilize the Tensor Cores. If you compare these car…
The 3090 RTX is also $1000 cheaper than the Titan, so there's that. It would be nice if there was a good way to express value per dollar. Perhaps in GLUE accuracy and training time.
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#76This is a poor comparison of performance. All of these networks are CNNs, and very old architectures at that. They are all probably memory bottlenecked which is why you see the consistent 50% improvement in FP32 perf. It is also not clear what batch sizes are being used for any of the tests. If you switch to FP16 training, you must increase the batch size to properly utilize the Tensor Cores. If you compare these car…
The 3090 RTX is also $1000 cheaper than the Titan, so there's that. It would be nice if there was a good way to express value per dollar. Perhaps in GLUE accuracy and training time.
Totally agree! I think 3090 could be a lot more cost effective for researchers to dabble with NLP. But it really grinds my gears when people post these misleading benchmarks... the 3090 is handicapped at half-rate tensor core performance while the Titan RTX is not.
So if you're someone who does their work mainly in FP32, you will see improved performance with the 3090. On the other hand, if you are an FP16 speed demon who needs to train GPT-3 over the weekend, stick with your Titans :)
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#77Earlier quoted context omitted.
At 50% utilization it beats the cloud in 2 months. 10% utilization, it beats the cloud within a year. If you're dabbling, definitely go with the cloud, but if you're turning around experiments on a regular basis, buying gets attractive quickly. And no, cards don't just keel over in a few months at 100%. Crypto miners ran that experiment. A typical card has years of 100% in it.
Only if your power is free (it isn't) and the machine the card is in is free (it isn't) and said machine produces no heat or noise (it does).
Power doesn't cost nearly as much for TCO to get anywhere near even the "preemptible price" of V100 (and probably A100 when it's ready) over a period of half a year. And now that 3090 has 24gb, which is needed for larger models, a solution with a couple of consumer cards is even more competitive for experimentation. You can also sell your cards in a year or two, and recover some of the costs. (All that of course if Nvidia's gimping of consumer cards doesn't significantly affect your code)
Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#78Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#79Re: Benchmarking TensorFlow on Nvidia GeForce RTX 3090
#80Earlier quoted context omitted.
At 50% utilization it beats the cloud in 2 months. 10% utilization, it beats the cloud within a year. If you're dabbling, definitely go with the cloud, but if you're turning around experiments on a regular basis, buying gets attractive quickly. And no, cards don't just keel over in a few months at 100%. Crypto miners ran that experiment. A typical card has years of 100% in it.
Only if your power is free (it isn't) and the machine the card is in is free (it isn't) and said machine produces no heat or noise (it does).
I rounded down AWS's price by $.06/hr, more than the price of electricity and cooling around here.