Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerated PyTorch Training on M1 Mac

pytorch.org

Post reply on HN
Print view View original

61–70 of 153 posts

Re: Accelerated PyTorch Training on M1 Mac

#61

Post by eklitzke »

iamaaditya wrote: post #57
sillyinseattle wrote: post #49

Earlier quoted context omitted.

Question about terminology (no background in AI). In econometrics, estimation is model fitting (training, I guess), and inference refers to hypothesis testing (e.g. t or F tests). What does inference mean here?

In machine learning (especially deep learning or neural networks), the 'training' is done by using Stochastic Gradient Descent. These gradients are computed using Backpropagation. Backpropagation requires you to do a backward pass of your model (typically many layers of neural weights) and thus requires you to keep in memory a lot of intermediate values (called activations). However, if you are doing "inference" that…

Just to add to this, the reason these inference accelerators have become big recently (see also the "neural core" in Pixel phones) is because they help doing inference tasks in real time (lower model latency) with better power usage than a GPU.

As a concrete example, on a camera you might want to run a facial detector so the camera can automatically adjust its focus when it sees a human face. Or you might want a person detector that can detect the outline of the person in the shot, so that you can blur/change their background in something like a Zoom call. All of these applications are going to work better if you can run your model at, say, 60 HZ instead of 20 HZ. Optimizing hardware to do inference tasks like this as fast as possible with the least possible power usage it pretty different from optimizing for all the things a GPU needs to do, so you might end up with hardware that has both and uses them for different tasks.

Re: Accelerated PyTorch Training on M1 Mac

#62

Post by my123 »

mkaic wrote: post #48

This is really cool for a number of reasons: 1.) Apple Silicon currently can't compete with Nvidia GPUs in terms of raw compute power, but they're already way ahead on energy efficiency. Training a small deep learning model on battery power on a laptop could actually be a thing now. Edit: I've been informed that for matrix math, Apple Silicon isn't actually ahead in efficiency 2.) Apple Silicon probably will compete…

> but they're already way ahead on energy efficiency

1) Nope. For neural network training not the case: https://tlkh.dev/benchmarking-the-apple-m1-max

And that's with the 3090 set at a very high 400W power limit, can get far more efficient when clocked lower.

(which is normal, because no dedicated matrix math accelerators on the GPU notably)

2) We'll see, hopefully Apple thinks that the market is worth bothering with... (which would be great)

3) Indeed, if you need a giant pool of VRAM above everything else at a relatively low price tag, Apple is indeed a quite enticing option. If you can stand Metal for your use case of course.

Re: Accelerated PyTorch Training on M1 Mac

#63

Post by my123 »

munro wrote: post #58
RicoElectrico wrote: post #42

Earlier quoted context omitted.

Most probably Neural Engine is optimized for inference, not training.

That /sounds/ right, but training still has a forward part, so OP does raise a really great question. And looking at the silicon, the neural engine is almost the size of the GPU. Really need someone educated in this area to chime in :)

The neural engine is only exposed through a CoreML inference API.

You can't even poke the ANE hardware directly from a regular process. The interface for accessing the neural engine is not hardened (you can easily crash the machine from it).

So the matter is essentially moot in practice as you'd need your users to run with SIP off...

Re: Accelerated PyTorch Training on M1 Mac

#64

Post by my123 »

smoldesu wrote: post #56
zdw wrote: post #53

Earlier quoted context omitted.

Everything old is new again - the M1 studio's unified memory echos the SGI O2 which had similar unified CPU/GPU memory back in the 90's. In both cases the unified memory machines outperformed much larger machines in specific use cases.

... specific use cases being the key operand here. Unified memory is cool, but there are reasons we don't use it at-scale: - It needs extremely high-bandwidth controllers, which severely limits the amount of memory you can use (Intel Macs could be configured with an order of magnitude more ram in it's server chips) - ECC is still off-the-table on M1 apparently - Most workloads aren't really constrained by memory acce…

> - It needs extremely high-bandwidth controllers, which severely limits the amount of memory you can use (Intel Macs could be configured with an order of magnitude more ram in it's server chips)

In the first half of 2023, NVIDIA Grace Superchip will ship with an 1TB memory config (930GB usable because ECC bits) on a 1024-bit wide LPDDR5X-8533 config (same width as M1 Ultra, with LPDDR5-6400).

So it's going to become much less of an issue really soon.

Re: Accelerated PyTorch Training on M1 Mac

#65

Post by brrrrrm »

my123 wrote: post #59
brrrrrm wrote: post #35

Earlier quoted context omitted.

have you actually benchmarked that? I think (someone please correct me if I'm way off here) the AMX instructions can hit ~2.8tflops (fp16) per co-processor and there are 2 on the 7-core M1. That's 5.6tflops vs the 4.6tflops the GPU can hit.

Yeah that's within the M1 family, but get within dGPUs and it doesn't even come close. 30Tflops for a 3080 for vector FP32, but 119Tflops FP16 dense with FP16 accumulate, 59.5 with FP32 accumulate, and if you exploit sparsity then that can go even higher.

Ah yes, I misunderstood your original comment

Re: Accelerated PyTorch Training on M1 Mac

#66

Post by sudosysgen »

mkaic wrote: post #48

This is really cool for a number of reasons: 1.) Apple Silicon currently can't compete with Nvidia GPUs in terms of raw compute power, but they're already way ahead on energy efficiency. Training a small deep learning model on battery power on a laptop could actually be a thing now. Edit: I've been informed that for matrix math, Apple Silicon isn't actually ahead in efficiency 2.) Apple Silicon probably will compete…

Apple Silicon is not ahead at all on energy efficiency for desktop workloads. If they were ahead on energy efficiency, they would simply be ahead on power. Indeed, GPUs are massively parallel architectures, and they are generally limited by the transistor and power budget (and memory, of course).

Apple is simply behind in the GPU space.

> At $4800, an M1 Ultra Mac Studio appears to be far and away the cheapest machine you can buy with 128GB of GPU memory. With proper PyTorch support, we'll actually be able to use this memory for training big models or using big batch sizes. For the kind of DL work I do where dataloading is much more of a bottleneck than actual raw compute power, Mac Studio is now looking very enticing.

The reason why it's cheaper is that its memory is at a fraction (around 20-35%) of the memory bandwidth of a 128GB equivalent GPU set up, which also has to be split with the CPU. This is an unavoidable bottleneck of shared memory systems, and for a great many applications this is a terminal performance bottleneck.

That's the reason you don't have a GPU with 128GB of normal DDR5. It would just be quite limited. Perhaps for some cases it can be useful.

Re: Accelerated PyTorch Training on M1 Mac

#67

Post by learndeeply »

jorgemf wrote: post #30
ekelsen wrote: post #10

Nice results! But why are people still reporting benchmark results on VGG? Does anybody actually use this network anymore? Better would be mobilenets or efficientNets or NFNets or vision transformers or almost anything that's come out in the 8 years since VGG was published (great work it was at the time!).

Probably because it will be impossible to compare with old results. If every year the community chooses a different model, how are you going to compare results year over year?

ResNets have been around for 7 years...

Re: Accelerated PyTorch Training on M1 Mac

#68

Post by ActorNightly »

mkaic wrote: post #48

This is really cool for a number of reasons: 1.) Apple Silicon currently can't compete with Nvidia GPUs in terms of raw compute power, but they're already way ahead on energy efficiency. Training a small deep learning model on battery power on a laptop could actually be a thing now. Edit: I've been informed that for matrix math, Apple Silicon isn't actually ahead in efficiency 2.) Apple Silicon probably will compete…

> but they're already way ahead on energy efficiency.

For raw compute like you need for ML training, the M1s efficiency doesn't matter. Under the hood at hardware level, you have a direct mapping of power consumption to compute circuit activation that you really can't get around.

The general efficiency of M1 is due its architecture and how it fits together with normal consumer use. Less stuff on the instruction decode, more efficient reordering, less energy wasted moving around data due to shared memory architecture, e.t.c

Re: Accelerated PyTorch Training on M1 Mac

#69

Post by Q6T46nT668w6i3m »

sillyinseattle wrote: post #49
RicoElectrico wrote: post #42

Earlier quoted context omitted.

Most probably Neural Engine is optimized for inference, not training.

Question about terminology (no background in AI). In econometrics, estimation is model fitting (training, I guess), and inference refers to hypothesis testing (e.g. t or F tests). What does inference mean here?

I’m surprised nobody has provided the basic explanation: inference, here, means matrix, matrix or matrix, scalar multiplication.

Re: Accelerated PyTorch Training on M1 Mac

#70

Post by Q6T46nT668w6i3m »

smoldesu wrote: post #56
zdw wrote: post #53

Earlier quoted context omitted.

Everything old is new again - the M1 studio's unified memory echos the SGI O2 which had similar unified CPU/GPU memory back in the 90's. In both cases the unified memory machines outperformed much larger machines in specific use cases.

... specific use cases being the key operand here. Unified memory is cool, but there are reasons we don't use it at-scale: - It needs extremely high-bandwidth controllers, which severely limits the amount of memory you can use (Intel Macs could be configured with an order of magnitude more ram in it's server chips) - ECC is still off-the-table on M1 apparently - Most workloads aren't really constrained by memory acce…

> Most workloads aren't really constrained by memory access in modern programs/kernels/compilers. Problems only show up when you want to run a GPU off the same memory, which is what these new Macs account for.

For sure but I expect this is different for the apps Apple _wants_ to write. It’s easy to imagine the next version of Logic or whatever doing fine tuning everywhere.

Post reply on HN