Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerated PyTorch Training on M1 Mac

pytorch.org

Post reply on HN
Print view View original

141–150 of 153 posts

Re: Accelerated PyTorch Training on M1 Mac

#141

Post by viraptor »

my123 wrote: post #63
munro wrote: post #58

Earlier quoted context omitted.

That /sounds/ right, but training still has a forward part, so OP does raise a really great question. And looking at the silicon, the neural engine is almost the size of the GPU. Really need someone educated in this area to chime in :)

The neural engine is only exposed through a CoreML inference API. You can't even poke the ANE hardware directly from a regular process. The interface for accessing the neural engine is not hardened (you can easily crash the machine from it). So the matter is essentially moot in practice as you'd need your users to run with SIP off...

That doesn't seem to be a huge issue. If someone actually does this for income, would they avoid disabling sip for 2x performance gain for example?

Re: Accelerated PyTorch Training on M1 Mac

#143

Post by derbOac »

iamaaditya wrote: post #57
sillyinseattle wrote: post #49

Earlier quoted context omitted.

Question about terminology (no background in AI). In econometrics, estimation is model fitting (training, I guess), and inference refers to hypothesis testing (e.g. t or F tests). What does inference mean here?

In machine learning (especially deep learning or neural networks), the 'training' is done by using Stochastic Gradient Descent. These gradients are computed using Backpropagation. Backpropagation requires you to do a backward pass of your model (typically many layers of neural weights) and thus requires you to keep in memory a lot of intermediate values (called activations). However, if you are doing "inference" that…

Worth noting that inference in "traditional" statistics and ML/AI/DL isn't really that different at some level. In both cases you have an inverse problem; in one case the parameters are about a group or population (e.g., something about all cats in existence), and in another it is about an individual case (something about a particular cat).

Re: Accelerated PyTorch Training on M1 Mac

#144

Post by tzekid »

singularity2001 wrote: post #142

The installation command generated on https://pytorch.org/get-started/locally/ didn't install the latest version for me. What did it was: pip3 install --pre torch==1.12.0.dev20220518 --extra-index-url https://download.pytorch.org/whl/nightly/cpu

Ahh just saw this after compiling pytorch from source. Thanks!

Re: Accelerated PyTorch Training on M1 Mac

#145

Post by [deleted] »

almostdigital wrote: post #129

Anyone actually got this to run on an M1 Mac? $ conda install pytorch torchvision torchaudio -c pytorch-nightly Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. PackagesNotFoundError: The fol…

[deleted]

Re: Accelerated PyTorch Training on M1 Mac

#146

Post by dekhn »

ray__ wrote: post #121
dekhn wrote: post #111

Earlier quoted context omitted.

it took me 20 years to learn this body of knowledge and now it can just sort of be summed up in a paragraph. When I learned and used gradient descent, you had to analytically determine your own gradients ( https://web.archive.org/web/20161028022707/https://genomics.... ). I went to grad school to learn how to determine my own gradients. Unfortunately, in my realm, loss landscapes have multiple minima, and gradient de…

What's your realm?

protein folding and structure prediction. Protein simulations typically define an energy function, similar to a loss function, over all the atoms in the protein. There are many terms: at least one per bonded atom pair, at least one per bonded atom triple, at least one per bonded atom quadruple, one per each non-bonded pair (although atoms that are distant can be excluded, sometimes making this a sparse matrix). If you start with a proposed model (say, random coordinates for all the atoms) and apply gradient descent, you'll end up with a mess. All those energy terms end up creating a high dimensional surface that is absurdly spiky in the details, and extremely wavy with many local minima at coarse grain.

Instead of using gradient descent, we used molecular dynamics (I'm unaware if this has a direct equivalent) to sample the space by moving along various isocontours (constant energy, or constant temp, or usually constant pressure). Even so, you have to do a lot of sampling- in my day, it was years of computer time, now it's months- to get a good approximation to the total landscape, and measure transition frequencies between areas of the landscape that correspond to energy barries (local maxima) that are smaller than the thermal energy avaialble to the system.

It's complicated. also, deep mind obviated all my work by providng that sequence data (which is cheap to obtain) can be used to predict very accurate structures with little or no simulation.

Re: Accelerated PyTorch Training on M1 Mac

#148

Post by munro »

my123 wrote: post #63
munro wrote: post #58

Earlier quoted context omitted.

That /sounds/ right, but training still has a forward part, so OP does raise a really great question. And looking at the silicon, the neural engine is almost the size of the GPU. Really need someone educated in this area to chime in :)

The neural engine is only exposed through a CoreML inference API. You can't even poke the ANE hardware directly from a regular process. The interface for accessing the neural engine is not hardened (you can easily crash the machine from it). So the matter is essentially moot in practice as you'd need your users to run with SIP off...

Sounds like you've you done a bit of digging around, you're efforts are appreciated. I found and a github of people sharing what they know, here's a guy live streaming hacking it and building a tinygrad https://youtu.be/mwmke957ki4

Re: Accelerated PyTorch Training on M1 Mac

#149

Post by ribit »

KingOfCoders wrote: post #139
blihp wrote: post #118

Earlier quoted context omitted.

How they do it at a conceptual level isn't a big secret: they don't need to minimize die area the way other companies do. For Apple, the die is just part of a chip that is part of the larger system they sell that they can amortize the cost over. nVidia doesn't have a system to do that with so their natural inclination is to lean towards keeping the die size as small as possible and just overclock the hell out of it.…

Also I thought Apple is adding a large slice of cache. When you look at the 3D-V cache on Ryzen for performance (+15%?), this has a large impact. And because they sell expensive stuff, they can afford to build expensive CPUs.

Cache doesn't matter for pure ALU efficiency though. I mean, I did tests on long dependent chains of FMAs, the only memory touched there are two internal registers.

Re: Accelerated PyTorch Training on M1 Mac

#150

Post by ribit »

blihp wrote: post #118
ribit wrote: post #110

Earlier quoted context omitted.

And yet somehow Apples GPU ALUs are more efficient at 3.8 watts per TFLOP. Mind, I am not talking about specialized matrix multiplication units that have a different internal organization and can do things like matrix multiplication much more efficiently, but about basic general-purpose GPU ALUs. The comparison of efficiency between Apple and Nvidia here is a bit misleading because one compares Apples general-purpose…

How they do it at a conceptual level isn't a big secret: they don't need to minimize die area the way other companies do. For Apple, the die is just part of a chip that is part of the larger system they sell that they can amortize the cost over. nVidia doesn't have a system to do that with so their natural inclination is to lean towards keeping the die size as small as possible and just overclock the hell out of it.…

But how does that apply to GPU ALUs? Looking at M1 die shots, they are comparatively tiny, and when comparing to other vendors, it doesn't seem like Apple is dedicated more logic space to the GPU. The M1 die is roughly 120mm2, an Nvidia Turing TU117 (GTX 1650) is roughly 200m2. Both feature the same amount of GPU ALUs (1024 32-bit units). And of course, M1's 5nm is around 5-6 times denser than Turing's 12nm, but M1 is an entire SoC with all kinds of components — not to mention a huge cache — the GPU takes maybe 20% of the die (let's say 1/3 if you also count in the display controller and memory controllers). All in all, the amount of normalised die space dedicated to GPU ALUs seems comparable.

Of course, my perspective here might be extremely naive, I know very little about semiconductor technology, just trying to understand the principal design differences.

Post reply on HN