Betteridge says no.
And Betteridge is wrong.
Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100
41–50 of 94 posts
Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100
#42"trainable_params 12,810" laughs (for comparison, GPT3: 175,000,000,000 parameters) Can Apple's M1 help you train tiny toy examples with no real-world relevance? You bet it can! Plus it looks like they are comparing Apples to Oranges ;) This seems to be 16 bit precision on the M1 and 32 bit on the V100. So the M1-trained model will most likely yield worse or unusable results, due to lack of precision. And lastly, the…
thanks for the thorough comment. the article is, unfortunately, just clickbait.
The comment is bogus empty snark (and factually wrong).
The arguments made (and I use the word arguments loosely):
"Too few trainable_params compared to GTP3".
GTP3 is several orders of magnitude higher than what people train, and so it's a useless comparison. It's like we're comparing a bike to an e-bike, and someone says "yeah, but can the e-bike run faster than a rocket?"
Second argument "Sure, it's faster than a machine that costs 3-4 fives more, but you should instead compare it to a machine that costs even more than that".
I can only take it as a troll comment.
Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100
#43Earlier quoted context omitted.
If you properly utilize your hardware, on premise (or colocation in an area with cheap electricity prices) is vastly cheaper and will likely continue to be for a while. I don't see how training models in the cloud makes financial sense for organizations that can utilize their hardware 24/7. For all others with burst workloads training in the cloud can make sense, but that has been the case for a while already.
We're not talking about organizations, though. I don't agree with your premise, either. People aren't training models 24/7, so the idea that it's "vastly cheaper and will continue to be for a while" isn't true.
> People aren’t training models 24/7
... uh, you sure about that? Let me go check on the 3 models I have concurrently training for my organization on 3 separate GPU servers (all 2 year old hardware to boot) that have been running continuously for the past 36 hours. It pretty much works out to 24/7 training for the past several months.
And BTW, this is massively cheaper for us than training in the cloud.
Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100
#44Earlier quoted context omitted.
If you properly utilize your hardware, on premise (or colocation in an area with cheap electricity prices) is vastly cheaper and will likely continue to be for a while. I don't see how training models in the cloud makes financial sense for organizations that can utilize their hardware 24/7. For all others with burst workloads training in the cloud can make sense, but that has been the case for a while already.
We're not talking about organizations, though. I don't agree with your premise, either. People aren't training models 24/7, so the idea that it's "vastly cheaper and will continue to be for a while" isn't true.
Instead of arguing back and forth, how about a test case instead?
Pretraining BERT takes 44 minutes on 1024 V100 GPUs [1]
This requires dedicated instances, since shared instances won't be able to get to peak performance if only because of the "noisy neighbour"-effect.
At GCP, a V100 costs $2.48/h [2], so Microsoft's experiment would've cost $2,539.52.
Smaller providers offer the same GPU at just $1.375/h [3], so a reasonable lower limit would be around $1,408.
For a single BERT pretraining, provided highly optimised workflows and distributed training scripts are already at hand, renting a GPU for single training tasks seems to be the way to go.
The cost of V100-equivalent end-user hardware (we don't need to run in a datacentre, dedicated workstations will do), is about $6,000 (e.g. a Quadro RTX 6000), provided you don't need double precision. The card will have equal FP32 performance, lower TGP and VRAM that sits between the 16 GB and 32 GB version of the V100.
Workstation hardware to go with such card will cost about $2,000, so $8,000 are a reasonable cost estimation. The cost of electricity varies between regions, but in the EU the average non-household price is about 0.13€/kWh [4].
Pretraining BERT therefore costs an estimated 1024 h * 0.13€/kWh * 0.5 kW ≈ 57€ in electricity (power consumption estimated from TGP + typical power consumptions of an Intel Xeon workstation from my own measurements when training models).
In order to get the break-even point we can use the following equation: t * $1,408 = $8,000 + t * $69, which results in t = 8,000/(1408-69) or t > 5.
In short, if you pretrain BERT 6 times, you safe money by BUYING a workstation and running it locally over renting cloud GPUs from a reasonably cheap provider.
This example only concerns BERT, but you can use the same reasoning for any model that you know the required compute time and VRAM requirements of.
This only concerns training, too - inference is a whole different can of worms entirely.
[1] https://www.deepspeed.ai/news/2020/05/27/fastest-bert-traini...
[2] https://cloud.google.com/compute/gpus-pricing
[3] https://www.exoscale.com/syslog/new-tesla-v100-gpu-offering/
[4] https://ec.europa.eu/eurostat/statistics-explained/index.php...
Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100
#45I categorize this as an exploration of how to benchmark desktop/workstation NPUs [1] similar to the exploration Daniel Lemire started with SIMD. Mobile SoC NPUs are used to deploy inference models on smartphones and IoT devices while discreet NPUs like Nvidia A100/V100 target cloud clusters. We don’t have apples-to-apples benchmarks like SPECint/SPECfp for the SoC accelerators in the M1 (GPU, NPU, etc.) so these earl…
> We don’t have apples-to-apples benchmarks
We do: https://mlperf.org/
Just run their benchmarks. Submitting your results there is a bit more complicated, because all results there are "verified" by independent entities.
If you feel like your AI use case is not well represented by any of the MLPerf benchmarks, open a discussion thread about it, propose a new benchmark, etc.
The set of benchmarks there increases all the time to cover new applications. For example, on top of the MLPerf Training and MLPerf Inference benchmark suites, we now have a new MLPerf HPC suite to capture ML of very large models.
Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100
#46Earlier quoted context omitted.
Yes and you'll see parameters tuned for V100, not parameters tuned for m1 somehow limping along on a V100 in emulation mode. I wouldn't complain about a benchmark executing any real world SOTA model on m1 and V100, but those will most likely not even run on the M1 due to memory constraints. So this article is like using an ios game to evaluate a Mac pro. You can do it, but it's not really useful.
You can count the number of GPUs having more than M1 memory(16 GB) in a single hand.
The M1 can only address 8 GB with its NPU/GPU.
Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100
#47"trainable_params 12,810" laughs (for comparison, GPT3: 175,000,000,000 parameters) Can Apple's M1 help you train tiny toy examples with no real-world relevance? You bet it can! Plus it looks like they are comparing Apples to Oranges ;) This seems to be 16 bit precision on the M1 and 32 bit on the V100. So the M1-trained model will most likely yield worse or unusable results, due to lack of precision. And lastly, the…
For the first graph: trainable parameters: 2236682
So it's a toy model...
Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100
#48Earlier quoted context omitted.
You would instead buy a used 1080 (no ti) for similar performance. The special thing about the V100 is that it's driver EULA allows data center usage. If you don't need that, there are other much cheaper options.
"Similar performance" still means 30%-50% slower [1] and half the RAM, not really that comparable. For much closer performance you should get a 2080ti, which should be roughly comparable in speed and have 11GB [edit: wrongly wrote 14GB before] of memory (against the 16GB for the V100). Price-wise you still save a lot of money, after quickly googling around, roughly $1200 vs. $15k-$20k. But you still lose something, e…
> But you still lose something, e.g. if you use half precision on V100 you get virtually double speed, if you do on a 1080 / 2080 you get... nothing because it's not supported.
That's not true. FP16 is supported and can be fast on 2080, although some frameworks fail to see the speed-up. I filed a bug report about this a year ago: https://github.com/apache/incubator-mxnet/issues/17665
What consumer GPUs lack is ECC and fast FP64.
Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100
#49When developing ML models, you rarely train "just one".
The article mentions that they explored a not-so-large hyper-parameter space (i.e. they trained multiple models with different parameters each).
It would be interesting to know how long does the whole process takes on the M1 vs the V100.
For the small models covered in the article, I'd guess that the V100 can train them all concurrently using MPS (multi-process service: multiple processes can concurrently use the GPU).
In particular it would be interesting to know, whether the V100 trains all models in the same time that it trains one, and whether the M1 does the same, or whether the M1 takes N times more time to train N models.
This could paint a completely different picture, particularly for the user perspective. When I go for lunch, coffee, or home, I usually spawn jobs training a large number of models, such that when I get back, all these models are trained.
I only start training a small number of models at the latter phases of development, when I have already explored a large part of the model space.
---
To make the analogy, what this article is doing is something similar to benchmarking a 64 core CPU against a 1 core CPU using a single threaded benchmark. The 64 core CPU happens to be slightly beefier and faster than the 1 core CPU, but it is more expensive and consumes more power because... it has 64x more cores. So to put things in perspective, it would make sense to also show a benchmark that can use 64x cores, which is the reason somebody would buy a 64-core CPU, and see how the single-core one compares (typically 64x slower).
---
To me, the only news here is that Apple GPU cores are not very far behind NVIDIA's cores for ML training, but there is much more to a GPGPU than just the perf that you get for small models in a small number of cores. Apple would still need to (1) catch up, and (2) extremely scale up their design. They probably can do both if they set their eyes on it. Exciting times.
Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100
#50>We can see better performance gains with the m1 when there are fewer weights to train likely due to the superior memory architecture of the M1. Wasn't this whole "M1 memory" thing decided to be a myth now some more technical people have dissected it?
I think two different memory things are being talked about:
1. There is an idea that M1 has RAM that is vastly higher bandwidth than intel/amd machines. In reality it is the same laptop ddr ram that other machines have, though at a very high clock rate. Not higher than the best intel laptops though. So the bandwidth is not any more amazing than a top end Intel laptop, and latency is no different.
2. But in this case I believe they are talking about the CPU and GPU both being able to freely access the same ram, as compared to a setup where you have a discrete GPU with it's own ram, where data must first be copied to the GPU ram for the GPU to do something with it. In some workloads this can be an inferior approach, in others it can be superior, as the GPU's ram is faster. The M1 model again isn't unique, as its similar to how game consoles work, I believe.