Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerated PyTorch Training on M1 Mac

pytorch.org

Post reply on HN
Print view View original

71–80 of 153 posts

Re: Accelerated PyTorch Training on M1 Mac

#71

Post by malshe »

sillyinseattle wrote: post #49
RicoElectrico wrote: post #42

Earlier quoted context omitted.

Most probably Neural Engine is optimized for inference, not training.

Question about terminology (no background in AI). In econometrics, estimation is model fitting (training, I guess), and inference refers to hypothesis testing (e.g. t or F tests). What does inference mean here?

I have background in both and it's very confusing to me. Inference in DL is running a trained model to predict/classify. Inference in stats and econometrics is totally different as you noted.

Re: Accelerated PyTorch Training on M1 Mac

#72

Post by dgacmu »

munro wrote: post #58
RicoElectrico wrote: post #42

Earlier quoted context omitted.

Most probably Neural Engine is optimized for inference, not training.

That /sounds/ right, but training still has a forward part, so OP does raise a really great question. And looking at the silicon, the neural engine is almost the size of the GPU. Really need someone educated in this area to chime in :)

You have to stash more information from the forward pass in order to calculate the gradients during backprop. You can't just naively use an inference accelerator as part of training - inference-only gets to discard intermediate activations immediately.

(Also, many inference accelerators use lower precision than you do when training)

There are tricks you can do to use inference to accelerate training, such as one we developed to focus on likely-poorly-performing examples: https://arxiv.org/abs/1910.00762

Re: Accelerated PyTorch Training on M1 Mac

#73

Post by p1esk »

sudosysgen wrote: post #66
mkaic wrote: post #48

This is really cool for a number of reasons: 1.) Apple Silicon currently can't compete with Nvidia GPUs in terms of raw compute power, but they're already way ahead on energy efficiency. Training a small deep learning model on battery power on a laptop could actually be a thing now. Edit: I've been informed that for matrix math, Apple Silicon isn't actually ahead in efficiency 2.) Apple Silicon probably will compete…

Apple Silicon is not ahead at all on energy efficiency for desktop workloads. If they were ahead on energy efficiency, they would simply be ahead on power. Indeed, GPUs are massively parallel architectures, and they are generally limited by the transistor and power budget (and memory, of course). Apple is simply behind in the GPU space. > At $4800, an M1 Ultra Mac Studio appears to be far and away the cheapest machin…

its memory is at a fraction (around 30-40%) of the memory bandwidth of a 128GB equivalent GPU setup

Here's some info about M1 memory bandwidth: https://www.anandtech.com/show/17024/apple-m1-max-performanc...

Re: Accelerated PyTorch Training on M1 Mac

#74

Post by sudosysgen »

p1esk wrote: post #73
sudosysgen wrote: post #66

Earlier quoted context omitted.

Apple Silicon is not ahead at all on energy efficiency for desktop workloads. If they were ahead on energy efficiency, they would simply be ahead on power. Indeed, GPUs are massively parallel architectures, and they are generally limited by the transistor and power budget (and memory, of course). Apple is simply behind in the GPU space. > At $4800, an M1 Ultra Mac Studio appears to be far and away the cheapest machin…

its memory is at a fraction (around 30-40%) of the memory bandwidth of a 128GB equivalent GPU setup Here's some info about M1 memory bandwidth: https://www.anandtech.com/show/17024/apple-m1-max-performanc...

Yes. And the M1 Ultra has even more memory bandwidth than the M1 Max. But a 128 GB system made of 3 NVidia A6000 has 3x768Gb/s of memory bandwidth, a more common AI-grade card has 2x2Tb/s of memory bandwidth, which simply dwarfs the M1 Ultra.

Re: Accelerated PyTorch Training on M1 Mac

#75

Post by smoldesu »

Q6T46nT668w6i3m wrote: post #70
smoldesu wrote: post #56

Earlier quoted context omitted.

... specific use cases being the key operand here. Unified memory is cool, but there are reasons we don't use it at-scale: - It needs extremely high-bandwidth controllers, which severely limits the amount of memory you can use (Intel Macs could be configured with an order of magnitude more ram in it's server chips) - ECC is still off-the-table on M1 apparently - Most workloads aren't really constrained by memory acce…

> Most workloads aren't really constrained by memory access in modern programs/kernels/compilers. Problems only show up when you want to run a GPU off the same memory, which is what these new Macs account for. For sure but I expect this is different for the apps Apple _wants_ to write. It’s easy to imagine the next version of Logic or whatever doing fine tuning everywhere.

What is there to fine-tune, in a program like Logic? I've often heard that word associated with using extended instruction sets and leveraging accelerators, but where would the M1 have "untapped power" so-to-speak? I don't think the "upgrade" from a CISC architecture to a RISC one can yield much opportunity for optimization, at least not besides what the compiler already does for you.

Re: Accelerated PyTorch Training on M1 Mac

#76

Post by paulmd »

WhitneyLand wrote: post #32
dagmx wrote: post #17

Earlier quoted context omitted.

Shaders are just the way compute is defined on the GPU. Why is that concerning to you?

It’s not the greatest term even for graphics only. People new to CG are likely to intuit “shaders” as something related to, well, shading, but vertex shaders et al have nothing to do with the color of a pixel or a polygon.

Wait until they learn a kernel has nothing to do with operating systems! And tensor operations have nothing to do with tensor objects! And texture memory often isn't even used for textures!

It's an unfortunate set of terminology due to the way this space evolved from graphics programming - shader cores used to do fixed-function shading! But then people wanted them to be able to run arbitrary shaders and not just fixed-function. And then hey, look at this neat processor, let's run a compute program on it. At first that was "compute shaders" running across graphics APIs, then came CUDA, and later OpenCL. But it is still running on the part of the hardware that provides shading to the graphics pipeline.

Similarly, texture memory actually used to be used for textures, now it is a general-purpose binding that coalesces any type of memory access that has 1D/2D/3D locality.

You kinda just get used to it. Lots of niches have their own lingo that takes some learning. Mathematics is incomprehensible without it, really.

Re: Accelerated PyTorch Training on M1 Mac

#77

Post by abm53 »

sillyinseattle wrote: post #49
RicoElectrico wrote: post #42

Earlier quoted context omitted.

Most probably Neural Engine is optimized for inference, not training.

Question about terminology (no background in AI). In econometrics, estimation is model fitting (training, I guess), and inference refers to hypothesis testing (e.g. t or F tests). What does inference mean here?

It is confusing that the ML community have come to use "inference" to mean prediction, whereas statisticians have long used it to refer to training/fitting, or hypothesis testing.

I'm not sure when or why this started.

Re: Accelerated PyTorch Training on M1 Mac

#78

Post by alfalfasprout »

mark_l_watson wrote: post #6
arecurrence wrote: post #5

This is much nicer ergonomics than what I had to do for tensorflow. It’s ostensibly out of the box support as a different torch device.

I agree. I appreciated the M1/Metal TensorFlow support, but that was not as easy to setup.

I mean, building tensorflow is generally an awful experience.

Re: Accelerated PyTorch Training on M1 Mac

#79

Post by hedgehog »

mkaic wrote: post #48

This is really cool for a number of reasons: 1.) Apple Silicon currently can't compete with Nvidia GPUs in terms of raw compute power, but they're already way ahead on energy efficiency. Training a small deep learning model on battery power on a laptop could actually be a thing now. Edit: I've been informed that for matrix math, Apple Silicon isn't actually ahead in efficiency 2.) Apple Silicon probably will compete…

To me the cool thing is working through a PyTorch-based course like FastAI on a local Mac may now be above the tolerably fast threshold.

Re: Accelerated PyTorch Training on M1 Mac

#80

Post by sillyinseattle »

eklitzke wrote: post #61
iamaaditya wrote: post #57

Earlier quoted context omitted.

In machine learning (especially deep learning or neural networks), the 'training' is done by using Stochastic Gradient Descent. These gradients are computed using Backpropagation. Backpropagation requires you to do a backward pass of your model (typically many layers of neural weights) and thus requires you to keep in memory a lot of intermediate values (called activations). However, if you are doing "inference" that…

Just to add to this, the reason these inference accelerators have become big recently (see also the "neural core" in Pixel phones) is because they help doing inference tasks in real time (lower model latency) with better power usage than a GPU. As a concrete example, on a camera you might want to run a facial detector so the camera can automatically adjust its focus when it sees a human face. Or you might want a pers…

Thank you @iamaaditya and @eklitzke . Very informative
Post reply on HN