Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerated PyTorch Training on M1 Mac

pytorch.org

Post reply on HN
Print view View original

81–90 of 153 posts

Re: Accelerated PyTorch Training on M1 Mac

#81

Post by mhh__ »

mkaic wrote: post #48

This is really cool for a number of reasons: 1.) Apple Silicon currently can't compete with Nvidia GPUs in terms of raw compute power, but they're already way ahead on energy efficiency. Training a small deep learning model on battery power on a laptop could actually be a thing now. Edit: I've been informed that for matrix math, Apple Silicon isn't actually ahead in efficiency 2.) Apple Silicon probably will compete…

The thing is with the efficiency (which I'm not sure of) and the competition (probably possible) is that the current nvidia lineup is pretty old and on an even older process. They have a big moat.

Re: Accelerated PyTorch Training on M1 Mac

#83

Post by mkaic »

sudosysgen wrote: post #66
mkaic wrote: post #48

This is really cool for a number of reasons: 1.) Apple Silicon currently can't compete with Nvidia GPUs in terms of raw compute power, but they're already way ahead on energy efficiency. Training a small deep learning model on battery power on a laptop could actually be a thing now. Edit: I've been informed that for matrix math, Apple Silicon isn't actually ahead in efficiency 2.) Apple Silicon probably will compete…

Apple Silicon is not ahead at all on energy efficiency for desktop workloads. If they were ahead on energy efficiency, they would simply be ahead on power. Indeed, GPUs are massively parallel architectures, and they are generally limited by the transistor and power budget (and memory, of course). Apple is simply behind in the GPU space. > At $4800, an M1 Ultra Mac Studio appears to be far and away the cheapest machin…

Interesting, I wasn't aware of the memory bandwidth point, though it makes sense. TIL!

Re: Accelerated PyTorch Training on M1 Mac

#84

Post by why_only_15 »

lekevicius wrote: post #36

Curiously neither PyTorch nor Tensorflow currently use M1's Neural Engine. Is too limited? Too hard to interact with? Not worth the effort?

The ANE only has support for calculations with fp16, int16 and int8 all of which are too small to train with (too much instability). A common thing to do is train in fp32 to be able to get the small differences and gradients and then once the model is frozen do inference on fp16 or bf16.

Re: Accelerated PyTorch Training on M1 Mac

#85

Post by jph00 »

why_only_15 wrote: post #84
lekevicius wrote: post #36

Curiously neither PyTorch nor Tensorflow currently use M1's Neural Engine. Is too limited? Too hard to interact with? Not worth the effort?

The ANE only has support for calculations with fp16, int16 and int8 all of which are too small to train with (too much instability). A common thing to do is train in fp32 to be able to get the small differences and gradients and then once the model is frozen do inference on fp16 or bf16.

Using mixed precision training you can do most operations in fp16 and just a few in fp32 where it's needed. This is the norm for NVIDIA GPU training nowadays. For instance using fastai add `.to_fp16()` after your learner call, and that happens automatically.

Re: Accelerated PyTorch Training on M1 Mac

#86

Post by dataexporter »

iamaaditya wrote: post #57
sillyinseattle wrote: post #49

Earlier quoted context omitted.

Question about terminology (no background in AI). In econometrics, estimation is model fitting (training, I guess), and inference refers to hypothesis testing (e.g. t or F tests). What does inference mean here?

In machine learning (especially deep learning or neural networks), the 'training' is done by using Stochastic Gradient Descent. These gradients are computed using Backpropagation. Backpropagation requires you to do a backward pass of your model (typically many layers of neural weights) and thus requires you to keep in memory a lot of intermediate values (called activations). However, if you are doing "inference" that…

This sounds really fascinating. Are there any resources that you'd recommend for someone who's starting out in learning all this? I'm a complete beginner when it comes to Machine Learning.

Re: Accelerated PyTorch Training on M1 Mac

#87

Post by matthew-wegner »

sudosysgen wrote: post #74
p1esk wrote: post #73

Earlier quoted context omitted.

its memory is at a fraction (around 30-40%) of the memory bandwidth of a 128GB equivalent GPU setup Here's some info about M1 memory bandwidth: https://www.anandtech.com/show/17024/apple-m1-max-performanc...

Yes. And the M1 Ultra has even more memory bandwidth than the M1 Max. But a 128 GB system made of 3 NVidia A6000 has 3x768Gb/s of memory bandwidth, a more common AI-grade card has 2x2Tb/s of memory bandwidth, which simply dwarfs the M1 Ultra.

For researchers, sure, but it's still quite an apples-to-oranges comparison.

A6000 is ~$5k per card. I guess you're referring to something like an A100 on that other spec, which is $10k/card (for 40GB of memory).

I do a fair bit of neural/AI art experimentation, where memory on the execution side is sometimes a limiting factor for me. I'm not training models, I'm not a hardcore researcher--those folks will absolutely be using NVIDIA's high-end stuff or TPU pods.

128GB in a Studio is super compelling if it means I can up-res some of my pieces without needing to use high-memory-but-super-slow CPU cloud VMs, or hope I get lucky with an A100 on Colab (or just pay for a GPU VM).

I have a 128GB/Ultra Studio in my office now. It's a great piece of kit, and a big reason I splurged on it--okay, maybe "excuse"--was that I expect it'll be useful for a lot of my side project workloads over the next couple of years...

Re: Accelerated PyTorch Training on M1 Mac

#88

Post by jph00 »

plonk wrote: post #27
ekelsen wrote: post #10

Nice results! But why are people still reporting benchmark results on VGG? Does anybody actually use this network anymore? Better would be mobilenets or efficientNets or NFNets or vision transformers or almost anything that's come out in the 8 years since VGG was published (great work it was at the time!).

> Does anybody actually use this network anymore? Why not? It's still good for simple classification tasks. We use it as an encoder for a segmentation model in some cases. Most ResNet variants are much heavier.

I don't think that's true - have a look at this analysis here:

https://www.kaggle.com/code/jhoward/which-image-models-are-b...

Those slow and inaccurate models at the bottom of the graph are the VGG models. A resnet34 is faster and more accurate than any VGG model. And there are better options now -- for example resnet34d is as fast as resnet34, and more accurate. And then convnext is dramatically better still.

Re: Accelerated PyTorch Training on M1 Mac

#89

Post by YetAnotherNick »

plonk wrote: post #27
ekelsen wrote: post #10

Nice results! But why are people still reporting benchmark results on VGG? Does anybody actually use this network anymore? Better would be mobilenets or efficientNets or NFNets or vision transformers or almost anything that's come out in the 8 years since VGG was published (great work it was at the time!).

> Does anybody actually use this network anymore? Why not? It's still good for simple classification tasks. We use it as an encoder for a segmentation model in some cases. Most ResNet variants are much heavier.

> ResNet > VGG: ResNet-50 is faster than VGG-16 and more accurate than VGG-19 (7.02 vs 9.0); ResNet-101 is about the same speed as VGG-19 but much more accurate than VGG-16 (6.21 vs 9.0).

https://github.com/jcjohnson/cnn-benchmarks#:~:text=ResNet%2....

Re: Accelerated PyTorch Training on M1 Mac

#90

Post by mattkrause »

sillyinseattle wrote: post #49
RicoElectrico wrote: post #42

Earlier quoted context omitted.

Most probably Neural Engine is optimized for inference, not training.

Question about terminology (no background in AI). In econometrics, estimation is model fitting (training, I guess), and inference refers to hypothesis testing (e.g. t or F tests). What does inference mean here?

Prediction.

The model is literally "inferring" something about its inputs: e.g., these pixels denote a hot dog, those don't.

Post reply on HN