Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100

wandb.ai

Post reply on HN
Print view View original

61–70 of 94 posts

Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100

#61

Post by tyingq »

vletal wrote: post #34
whywhywhywhy wrote: post #15

>We can see better performance gains with the m1 when there are fewer weights to train likely due to the superior memory architecture of the M1. Wasn't this whole "M1 memory" thing decided to be a myth now some more technical people have dissected it?

Could you please provide some resources on how the unified memory model supposedly works? Why is it a "myth"?

I believe it's referring not to unified memory, but some speculation that the memory being closer to the CPU makes some notable difference. That line was in a fair amount of the initial articles about the M1, and fits the "myth" description.

Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100

#62

Post by lldbg »

procrastinatus wrote: post #60

One thing I haven’t seen much mention of is getting things to run on the M1’s neural engine instead of the GPU - it seems like the neural engine has ~3x more compute capacity and is specifically optimized for this type of computation. Has anyone spotted any work allowing a mainstream tensor library (e.g. jax, tf, pytorch) to run on the neural engine?

George hotz got his "for play" tensor library[a] to run on the Apple Neural Engine (ANE). The results were somewhat dissapointing, however, and currently it only does relu.

[a]: https://github.com/geohot/tinygrad

Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100

#63

Post by volta87 »

nightcracker wrote: post #56
volta87 wrote: post #49

When developing ML models, you rarely train "just one". The article mentions that they explored a not-so-large hyper-parameter space (i.e. they trained multiple models with different parameters each). It would be interesting to know how long does the whole process takes on the M1 vs the V100. For the small models covered in the article, I'd guess that the V100 can train them all concurrently using MPS (multi-process…

> When developing ML models, you rarely train "just one". Depends on your field. In Reinforcement Learning you often really do train just one , at least on the same data set (since the data set often is dynamically generated based on the behavior of the previous iteration of the model).

Even in reinforcement learning you can train multiple model with different data-sets concurrently and combine them for the next iteration.

Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100

#65

Post by sradman »

volta87 wrote: post #45
sradman wrote: post #35

I categorize this as an exploration of how to benchmark desktop/workstation NPUs [1] similar to the exploration Daniel Lemire started with SIMD. Mobile SoC NPUs are used to deploy inference models on smartphones and IoT devices while discreet NPUs like Nvidia A100/V100 target cloud clusters. We don’t have apples-to-apples benchmarks like SPECint/SPECfp for the SoC accelerators in the M1 (GPU, NPU, etc.) so these earl…

> We don’t have apples-to-apples benchmarks We do: https://mlperf.org/ Just run their benchmarks. Submitting your results there is a bit more complicated, because all results there are "verified" by independent entities. If you feel like your AI use case is not well represented by any of the MLPerf benchmarks, open a discussion thread about it, propose a new benchmark, etc. The set of benchmarks there increases all t…

> on top of the MLPerf Training and MLPerf Inference benchmark suites, we now have a new MLPerf HPC suite to capture ML of very large models.

I think the challenge is selecting the tests that best represent the typical ML/DL use cases for the M1 and comparing it to an alternative such as the V100 using a common toolchain like Tensorflow. One of the problems that I see is that the optimizer/codegen of the toolchain is a key component; the M1 has both GPU and Neural Engine and we don’t know which accelerator is targeted or even possibly both. Should we benchmark ML Create on M1 vs A14 or A12X? Perhaps it is my ignorance but I don’t think we are at a point where our existing benchmarks can be applied meaningfully with the M1 but I’m sure we will get there soon.

Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100

#66

Post by rsynnott »

whywhywhywhy wrote: post #15

>We can see better performance gains with the m1 when there are fewer weights to train likely due to the superior memory architecture of the M1. Wasn't this whole "M1 memory" thing decided to be a myth now some more technical people have dissected it?

As with many things, there isn't one "M1 memory" thing. It's a combination of myth and real stuff. No, it isn't ultra-low latency or high-bandwidth. But on the other hand, single core achievable bandwidth is very high.

Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100

#67

Post by dragontamer »

gameswithgo wrote: post #50
whywhywhywhy wrote: post #15

>We can see better performance gains with the m1 when there are fewer weights to train likely due to the superior memory architecture of the M1. Wasn't this whole "M1 memory" thing decided to be a myth now some more technical people have dissected it?

I think two different memory things are being talked about: 1. There is an idea that M1 has RAM that is vastly higher bandwidth than intel/amd machines. In reality it is the same laptop ddr ram that other machines have, though at a very high clock rate. Not higher than the best intel laptops though. So the bandwidth is not any more amazing than a top end Intel laptop, and latency is no different. 2. But in this case…

> In reality it is the same laptop ddr ram that other machines have

LPDDR4 is more well known for cell phones than laptops actually. I think it shows the stagnation of the laptop market (and DDR4) that LPDDR4 is really catching up (and then some). Or maybe... because cell phones are more widespread these days, cell phones just naturally get the better tech?

On the other hand, M1 is pretty wide. Apple clearly is tackling the memory bottleneck very strongly in its design.

DDR5 is going to be the next major step forward for desktops/laptops.

> 2. But in this case I believe they are talking about the CPU and GPU both being able to freely access the same ram, as compared to a setup where you have a discrete GPU with it's own ram, where data must first be copied to the GPU ram for the GPU to do something with it. In some workloads this can be an inferior approach, in others it can be superior, as the GPU's ram is faster. The M1 model again isn't unique, as its similar to how game consoles work, I believe.

More than just the "same RAM", but probably even shares the same last-level cache. Both AMD's chips and Intel's iGPUs share the cache with its CPU/GPU hybrid architectures.

However: it seems like on-core SIMD units (aka: AVX or ARM NEON / SVE) are even lower latency, since those share L1 cache.

Any situation where you need low latency but SIMD, it makes more sense to use AVX / SVE than even waiting for L3 cache to talk to the iGPU. Any situation where you need massive parallelism, a dedicated 3090 is more useful.

Its going to be tough to figure out a good use of iGPUs: they're being squeezed on the latency front (by things like A64Fx: 512-bit ARM SIMD, as well as AVX512 on the Intel side), and also squeezed by the bandwidth front (by classic GPUs)

Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100

#68

Post by rrss »

littlestymaar wrote: post #19
fxtentacle wrote: post #13

Earlier quoted context omitted.

You would instead buy a used 1080 (no ti) for similar performance. The special thing about the V100 is that it's driver EULA allows data center usage. If you don't need that, there are other much cheaper options.

> The special thing about the V100 is that it's driver EULA allows data center usage. Wait what? Is it the only thing? That sounds hard to believe: if true, using the open driver (Nouveau) instead of Nvidia's proprietary one would be a massive money saver for datacenters operators (and even if Nouveau doesn't support the features you'd want already, supporting their development would be much cheaper for a company lik…

No, that's not the only thing.

Other characteristics of V100 that may be interesting to people buying GPUs for data centers:

- higher capacity GPU memory. 1080 has 8 GB, V100 has 16 or 32 GB.

- higher bandwidth GPU memory. V100 has HBM2 with a peak of 900 GB/s, 1080 has G5X with a peak of ~300 GB/s.

- ECC support.

- data center certification + warranty

(The geforce warranty covers normal consumer usage, like gaming, and does not cover datacenter use)

- availability of enterprise support contracts.

(If you are buying a ton of GPUs to put in a datacenter, you probably don't want to end up on the normal consumer support line when something goes wrong)

- fast fp64

There are probably others

Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100

#69

Post by breuleux »

fxtentacle wrote: post #30
FeepingCreature wrote: post #28

Earlier quoted context omitted.

How does AMD stuff like Radeon VII or MI100 hold up?

Can't use it because most AI frameworks won't run on AMD because they did not implement suitable back-ends (yet).

There's one for PyTorch, I tested it about a year ago. You have to compile it from scratch and IIRC it translates/compile CUDA to ROCm at runtime which causes noticeable pauses on the first run. There may be other tweaks you have to do too. Once set up it performs decently, though.

Re: Analyzing the performance of Tensorflow training on M1 Mac Mini and Nvidia V100

#70

Post by tbalsam »

This is on a model designed to run faster on CPUs. It's like dropping a bowling ball on your foot and claiming excitement that you feel bruised after a few days.

Maybe there's something interesting there, definitely, but the overhype of the title takes away any significant amount of clout I'd give to the publishers for research. If you find something interesting, say it, and stop making vapid generalizations for the sake of more clicks.

Remember, we only can feed the AI hype bubble when we do this. It might be good results, but we need to be at least realistic about it, or there won't be an economy of innovation for people to listen to in the future, because they've tuned it out with all of the crap marketing that comes/came before it.

Thanks for coming to my TED Talk!

Post reply on HN