Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerated PyTorch Training on M1 Mac

pytorch.org

Post reply on HN
Print view View original

51–60 of 153 posts

Re: Accelerated PyTorch Training on M1 Mac

#51

Post by upwardbound »

sillyinseattle wrote: post #49
RicoElectrico wrote: post #42

Earlier quoted context omitted.

Most probably Neural Engine is optimized for inference, not training.

Question about terminology (no background in AI). In econometrics, estimation is model fitting (training, I guess), and inference refers to hypothesis testing (e.g. t or F tests). What does inference mean here?

Inference here means "running" the model. So maybe it has a similar meaning as in econometrics?

Training is learning the weights (millions or billions of parameters) that control the model's behavior, vs inference is "running" the trained model on user data.

Re: Accelerated PyTorch Training on M1 Mac

#53

Post by zdw »

buildbot wrote: post #46

This is very interesting since the M1 studio supports 128GB of unified memory - training a large memory heavy model slowly on a single device could be interesting, or inferencing a very large model.

Everything old is new again - the M1 studio's unified memory echos the SGI O2 which had similar unified CPU/GPU memory back in the 90's.

In both cases the unified memory machines outperformed much larger machines in specific use cases.

Re: Accelerated PyTorch Training on M1 Mac

#54

Post by ivstitia »

Scene_Cast2 wrote: post #8

I'm curious about the performance compared to something like, say, the RTX 3070.

Here are some comparison numbers I've come across: https://wandb.ai/tcapelle/apple_m1_pro/reports/Deep-Learning...

It is not really comparable on a step per second level but the power consumption and now GPU memory will make it pretty enticing.

Re: Accelerated PyTorch Training on M1 Mac

#55

Post by munro »

yess! This is important for me, because I don't have any $$$ to rent GPUs for personal projects. Now we just need M1 support for JAX.

Since there are no hard benchmarks against other GPUs, here's a Geekbench against an RTX 3080 Mobile laptop I have [1]. Looks like it's about 2x slower--the RTX laptop absolutely rips for gaming, I love it.

[1] https://browser.geekbench.com/v5/compute/compare/4140651?bas...

Re: Accelerated PyTorch Training on M1 Mac

#56

Post by smoldesu »

zdw wrote: post #53
buildbot wrote: post #46

This is very interesting since the M1 studio supports 128GB of unified memory - training a large memory heavy model slowly on a single device could be interesting, or inferencing a very large model.

Everything old is new again - the M1 studio's unified memory echos the SGI O2 which had similar unified CPU/GPU memory back in the 90's. In both cases the unified memory machines outperformed much larger machines in specific use cases.

...specific use cases being the key operand here. Unified memory is cool, but there are reasons we don't use it at-scale:

- It needs extremely high-bandwidth controllers, which severely limits the amount of memory you can use (Intel Macs could be configured with an order of magnitude more ram in it's server chips)

- ECC is still off-the-table on M1 apparently

- Most workloads aren't really constrained by memory access in modern programs/kernels/compilers. Problems only show up when you want to run a GPU off the same memory, which is what these new Macs account for.

- Most of the so-called "specific workloads" that you're outlining aren't very general applications. So far I've only seen ARM outrun x86 in some low-precision physics demos, which is... fine, I guess? I still don't foresee meteorologists dropping their Intel rigs to buy a Mac Studio anytime soon.

Re: Accelerated PyTorch Training on M1 Mac

#57

Post by iamaaditya »

sillyinseattle wrote: post #49
RicoElectrico wrote: post #42

Earlier quoted context omitted.

Most probably Neural Engine is optimized for inference, not training.

Question about terminology (no background in AI). In econometrics, estimation is model fitting (training, I guess), and inference refers to hypothesis testing (e.g. t or F tests). What does inference mean here?

In machine learning (especially deep learning or neural networks), the 'training' is done by using Stochastic Gradient Descent. These gradients are computed using Backpropagation. Backpropagation requires you to do a backward pass of your model (typically many layers of neural weights) and thus requires you to keep in memory a lot of intermediate values (called activations). However, if you are doing "inference" that is if the goal is only to get the result but not improve the model, then you don't have to do the backpropagation and thus you don't need to store/save the intermediate values. As the layers and number of parameters in Deep Learning grows, this difference in computation in training vs inference becomes signifiant. In most modern applications of ML, you train once but infer many times, and thus it makes sense to have specialized hardware that is optimized for "inference" at the cost of its inability to do "training".

Re: Accelerated PyTorch Training on M1 Mac

#58

Post by munro »

RicoElectrico wrote: post #42
lekevicius wrote: post #36

Curiously neither PyTorch nor Tensorflow currently use M1's Neural Engine. Is too limited? Too hard to interact with? Not worth the effort?

Most probably Neural Engine is optimized for inference, not training.

That /sounds/ right, but training still has a forward part, so OP does raise a really great question. And looking at the silicon, the neural engine is almost the size of the GPU. Really need someone educated in this area to chime in :)

Re: Accelerated PyTorch Training on M1 Mac

#59

Post by my123 »

brrrrrm wrote: post #35
my123 wrote: post #23

Earlier quoted context omitted.

AMX is indeed very nice for FP64 where customer GPUs aren't an alternative at all. However, for lower precisions (which is what deep learning uses), you're much better off with a GPU.

have you actually benchmarked that? I think (someone please correct me if I'm way off here) the AMX instructions can hit ~2.8tflops (fp16) per co-processor and there are 2 on the 7-core M1. That's 5.6tflops vs the 4.6tflops the GPU can hit.

Yeah that's within the M1 family, but get within dGPUs and it doesn't even come close.

30Tflops for a 3080 for vector FP32, but 119Tflops FP16 dense with FP16 accumulate, 59.5 with FP32 accumulate, and if you exploit sparsity then that can go even higher.

Re: Accelerated PyTorch Training on M1 Mac

#60

Post by smoldesu »

mkaic wrote: post #48

This is really cool for a number of reasons: 1.) Apple Silicon currently can't compete with Nvidia GPUs in terms of raw compute power, but they're already way ahead on energy efficiency. Training a small deep learning model on battery power on a laptop could actually be a thing now. Edit: I've been informed that for matrix math, Apple Silicon isn't actually ahead in efficiency 2.) Apple Silicon probably will compete…

There's definitely competition, and it's going to be really interesting to watch Nvidia and Apple duke it out over the next few years:

- Apple undoubtedly owns the densest nodes, and will fight TSMC tooth-and-nail over first dibs on whatever silicon they have coming next.

- Apple's current GPU design philosophy relies on horizontally scaling the tech they already use, whereas Nvidia has been scaling vertically, albeit slowly.

- Nvidia has insane engineers. Despite the fact they're using silicon that's more than twice as large by-area when compared to Apple, they're still doubling their numbers across the board. And that's their last-gen tech too, the comparison once they're on 5nm later this summer is going to be insane.

I expect things to be very heated by the end of this year, with new Nvidia, Intel and potentially new Apple GPUs.

Post reply on HN