Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerated PyTorch Training on M1 Mac

pytorch.org

Post reply on HN
Print view View original

101–110 of 153 posts

Re: Accelerated PyTorch Training on M1 Mac

#101

Post by jorgemf »

learndeeply wrote: post #67
jorgemf wrote: post #30

Earlier quoted context omitted.

Probably because it will be impossible to compare with old results. If every year the community chooses a different model, how are you going to compare results year over year?

ResNets have been around for 7 years...

It doesn't matter. Deep learning have been mainstream for only 10 years. MNIST is a dataset from 1998 and it is still being used in research papers. The most important thing is to have a constant baseline, and ResNets are a baseline.

Think about changing the model every other year: - 2015: ResNet trained in Nvidia k80 - 2017: Inception trained in Nvidia 1080 ti - 2019: Transformer trained in Nvidia V100 - 2021: GTP-3 trained in a cluster

Now you have your new fancy algorithm X and an Nvidia 4090. How much better is your algorithm compared to the state of the art, and how much have you improved compared to the algorithms 5 years ago? Now you are in a nightmare and you have to run all the past algorithms in order to compare it. Or how fast is the new Nvidia card? which noone still have and nvidia has decided to give numbers based on a their own model?

Re: Accelerated PyTorch Training on M1 Mac

#102

Post by dekhn »

mkaic wrote: post #48

This is really cool for a number of reasons: 1.) Apple Silicon currently can't compete with Nvidia GPUs in terms of raw compute power, but they're already way ahead on energy efficiency. Training a small deep learning model on battery power on a laptop could actually be a thing now. Edit: I've been informed that for matrix math, Apple Silicon isn't actually ahead in efficiency 2.) Apple Silicon probably will compete…

I remain skeptical that Apple's best GPU silicon will match nvidia's premiere products (either the top-end desktop card, or a server monster) for training.

It seems like this is ideal as an accelerator for already trained models; one can imagine Photoshop utilizing it for deep-learning based infill-painting.

I was doing training on battery with a laptop that had a 1080 and could do training; I have trained models on the airplane while totalyl unplugged and still had enough power to websurf afterwards.

Re: Accelerated PyTorch Training on M1 Mac

#103

Post by sbeckeriv »

mrchucklepants wrote: post #9
sbeckeriv wrote: post #4

What is the * in the chart referencing?

Probably supposed to be referencing the text under the plot stating the specific configuration of the hardware and software.

looks like the website was updated after I posted. I used page search to look for the *.

Re: Accelerated PyTorch Training on M1 Mac

#104

Post by mkaic »

sudosysgen wrote: post #94
matthew-wegner wrote: post #87

Earlier quoted context omitted.

For researchers, sure, but it's still quite an apples-to-oranges comparison. A6000 is ~$5k per card. I guess you're referring to something like an A100 on that other spec, which is $10k/card (for 40GB of memory). I do a fair bit of neural/AI art experimentation, where memory on the execution side is sometimes a limiting factor for me. I'm not training models, I'm not a hardcore researcher--those folks will absolutely…

Hmm, that's interesting. What kind of inference workload requires more than the 48GB of memory you'd get from 2 3090s, for example? I'm genuinely curious because I haven't ran across them and it sounds interesting

Not sure about inference but for training, 128GB is big enough to fit a decent-sized dataset entirely into memory, which causes a massive speedup. It's also probably cheaper to get a 128GB Mac Studio than a dual-3090 rig unless you're willing to build the rig yourself and pay the bare minimum for every component except the GPUs themselves.

As for 128GB memory on-inference models that a consumer would be interested in, I got nothing, though it certainly seems like it would be fun to mess around with haha

Re: Accelerated PyTorch Training on M1 Mac

#105

Post by zdw »

my123 wrote: post #64
smoldesu wrote: post #56

Earlier quoted context omitted.

... specific use cases being the key operand here. Unified memory is cool, but there are reasons we don't use it at-scale: - It needs extremely high-bandwidth controllers, which severely limits the amount of memory you can use (Intel Macs could be configured with an order of magnitude more ram in it's server chips) - ECC is still off-the-table on M1 apparently - Most workloads aren't really constrained by memory acce…

> - It needs extremely high-bandwidth controllers, which severely limits the amount of memory you can use (Intel Macs could be configured with an order of magnitude more ram in it's server chips) In the first half of 2023, NVIDIA Grace Superchip will ship with an 1TB memory config (930GB usable because ECC bits) on a 1024-bit wide LPDDR5X-8533 config (same width as M1 Ultra, with LPDDR5-6400). So it's going to become…

> So it's going to become much less of an issue really soon.

The main issue would be trying to purchase one of those, which is likely going to be both very rare and orders of magnitude more expensive than a Mac Studio.

The Mac Studio isn't some crazy exotic hardware like datacenter class GPUs, but definitely has some exotic capabilities.

Re: Accelerated PyTorch Training on M1 Mac

#106

Post by matthew-wegner »

sudosysgen wrote: post #94
matthew-wegner wrote: post #87

Earlier quoted context omitted.

For researchers, sure, but it's still quite an apples-to-oranges comparison. A6000 is ~$5k per card. I guess you're referring to something like an A100 on that other spec, which is $10k/card (for 40GB of memory). I do a fair bit of neural/AI art experimentation, where memory on the execution side is sometimes a limiting factor for me. I'm not training models, I'm not a hardcore researcher--those folks will absolutely…

Hmm, that's interesting. What kind of inference workload requires more than the 48GB of memory you'd get from 2 3090s, for example? I'm genuinely curious because I haven't ran across them and it sounds interesting

Mostly it's old-school transfer style transfer! Well, "old" in the sense that it's pre-CLIP. I've played with CLIP-guided stuff too, but I've been tinkering with a custom style transfer workflow for a few years. The pipeline here is fractal IFS images (Chaotica/JWildfire) -> misc processing -> style transfer -> photo editing, basically.

Only the workflow is the custom part--the core here is literally the original jcjohnson implementation. Occasionally I look around at recent work in the area, but most seems focused on fast (video-speed) inference or pre-baked style models. I've never seen something that retains artistic flexibility.

My original gut feeling on style transfer was that it would be possible to mold it into a neat tool, but most people bumped into it, ran their profile photo against Starry Night, said "cool" and bounced off. And I get that--parameter tuning can be a sloooow process. When I really explore a series with a particular style I start to feed it custom content images made just for how it's reacting with various inputs.

Here's a piece that just finished a few minutes ago: https://mwegner.com/misc/styled_render-BMrHXWz_2RBaUq8pAYKfL...

That's from a local server in my garage with a K80. At some point I had two K80s in there (so basically four K40s with how they work), but dialed it back for power consumption/power reasons.

I do have a 3090 in the house, and a decent amount of cloud infra that I sometimes tap. The jcjohnson implementation is so far back that it doesn't even run against modern hardware. At some point I need to sort that out, or figure out how to wrangle a more modern implementation into behaving in the way that I like.

I don't really post these anywhere, although do throw them over the wall on Twitter if anyone is curious to see more. These are a mix of things, although the CLIP/Midjourney/etc stuff is pretty easy to spot: https://twitter.com/mwegner/media

Re: Accelerated PyTorch Training on M1 Mac

#107

Post by my123 »

zdw wrote: post #105
my123 wrote: post #64

Earlier quoted context omitted.

> - It needs extremely high-bandwidth controllers, which severely limits the amount of memory you can use (Intel Macs could be configured with an order of magnitude more ram in it's server chips) In the first half of 2023, NVIDIA Grace Superchip will ship with an 1TB memory config (930GB usable because ECC bits) on a 1024-bit wide LPDDR5X-8533 config (same width as M1 Ultra, with LPDDR5-6400). So it's going to become…

> So it's going to become much less of an issue really soon. The main issue would be trying to purchase one of those, which is likely going to be both very rare and orders of magnitude more expensive than a Mac Studio. The Mac Studio isn't some crazy exotic hardware like datacenter class GPUs, but definitely has some exotic capabilities.

> The Mac Studio isn't some crazy exotic hardware like datacenter class GPUs, but definitely has some exotic capabilities.

Datacenter class GPUs are expensive yeah, but are quite easy to buy, even in a single unit amount.

example: https://www.dell.com/en-us/work/shop/nvidia-ampere-a100-pcie... for the first random link, but there are other stores selling them for significantly cheaper.

I wonder what their CPU pricing will be though... we'll see I guess.

Re: Accelerated PyTorch Training on M1 Mac

#108

Post by sudosysgen »

smlacy wrote: post #100
sudosysgen wrote: post #40

Earlier quoted context omitted.

It compares unfavourably, but then again NVidia GPUs on laptop are massive powerhogs.

Do apple users really require the ability to train large ML models while mobile and without access to A/C power? Is this a real-world use case for the target market?

Indeed, I doubt anyone really needs that. And anyways while training a model you'd be lucky to get an hour of battery life even on an M1 Max.

Re: Accelerated PyTorch Training on M1 Mac

#109

Post by h-jones »

omegalulw wrote: post #99
jph00 wrote: post #85

Earlier quoted context omitted.

Using mixed precision training you can do most operations in fp16 and just a few in fp32 where it's needed. This is the norm for NVIDIA GPU training nowadays. For instance using fastai add `.to_fp16()` after your learner call, and that happens automatically.

How is the choice between fp16 and fp32 made? Is it like if any gradients in the tensor need the extra range you use fp32?

The PyTorch docs give a pretty good overview of AMP here https://pytorch.org/tutorials/recipes/recipes/amp_recipe.htm... and an overview of which operations cast to which dtype can be found here https://pytorch.org/docs/stable/amp.html#autocast-op-referen....

Edit: Fixed second link.

Re: Accelerated PyTorch Training on M1 Mac

#110

Post by ribit »

ActorNightly wrote: post #68
mkaic wrote: post #48

This is really cool for a number of reasons: 1.) Apple Silicon currently can't compete with Nvidia GPUs in terms of raw compute power, but they're already way ahead on energy efficiency. Training a small deep learning model on battery power on a laptop could actually be a thing now. Edit: I've been informed that for matrix math, Apple Silicon isn't actually ahead in efficiency 2.) Apple Silicon probably will compete…

> but they're already way ahead on energy efficiency. For raw compute like you need for ML training, the M1s efficiency doesn't matter. Under the hood at hardware level, you have a direct mapping of power consumption to compute circuit activation that you really can't get around. The general efficiency of M1 is due its architecture and how it fits together with normal consumer use. Less stuff on the instruction decod…

And yet somehow Apples GPU ALUs are more efficient at 3.8 watts per TFLOP. Mind, I am not talking about specialized matrix multiplication units that have a different internal organization and can do things like matrix multiplication much more efficiently, but about basic general-purpose GPU ALUs.

The comparison of efficiency between Apple and Nvidia here is a bit misleading because one compares Apples general-purpose ALUs to Nvidia’s specialized ALUs. For a more direct efficiency comparison, one would need to compare the Tensor Cores against the AMX or ANE coprocessors.

As to how Apple achieves such high efficiency, nobody knows. The fact that they are on 5nm node might help, but there must be something special about the ALU design as well. My speculation is that they are wider and much more simpler than in other GPUs, which directly translates to efficiency wins.

Post reply on HN