Accelerated PyTorch Training on M1 Mac

Post by ray__ » Thu, May 19, 2022, 12:59 AM UTC

dekhn wrote: post #111

iamaaditya wrote: post #57

Earlier quoted context omitted.

In machine learning (especially deep learning or neural networks), the 'training' is done by using Stochastic Gradient Descent. These gradients are computed using Backpropagation. Backpropagation requires you to do a backward pass of your model (typically many layers of neural weights) and thus requires you to keep in memory a lot of intermediate values (called activations). However, if you are doing "inference" that…

it took me 20 years to learn this body of knowledge and now it can just sort of be summed up in a paragraph. When I learned and used gradient descent, you had to analytically determine your own gradients ( https://web.archive.org/web/20161028022707/https://genomics.... ). I went to grad school to learn how to determine my own gradients. Unfortunately, in my realm, loss landscapes have multiple minima, and gradient de…

Post by sanxiyn » Thu, May 19, 2022, 1:33 AM UTC

6gvONxR4sf7o wrote: post #24

ekelsen wrote: post #10

Nice results! But why are people still reporting benchmark results on VGG? Does anybody actually use this network anymore? Better would be mobilenets or efficientNets or NFNets or vision transformers or almost anything that's come out in the 8 years since VGG was published (great work it was at the time!).

> But why are people still reporting benchmark results on VGG? It makes me feel like i’m missing something! Is is still used as a backbone in the same way as legacy code is everywhere, or is it something else entirely??

Post by sshlocalhost98 » Thu, May 19, 2022, 2:17 AM UTC

iamaaditya wrote: post #57

sillyinseattle wrote: post #49

Earlier quoted context omitted.

Question about terminology (no background in AI). In econometrics, estimation is model fitting (training, I guess), and inference refers to hypothesis testing (e.g. t or F tests). What does inference mean here?

In machine learning (especially deep learning or neural networks), the 'training' is done by using Stochastic Gradient Descent. These gradients are computed using Backpropagation. Backpropagation requires you to do a backward pass of your model (typically many layers of neural weights) and thus requires you to keep in memory a lot of intermediate values (called activations). However, if you are doing "inference" that…

Post by [deleted] » Thu, May 19, 2022, 3:05 AM UTC

highfrequency wrote: post #124

my123 wrote: post #62

Earlier quoted context omitted.

> but they're already way ahead on energy efficiency 1) Nope. For neural network training not the case: https://tlkh.dev/benchmarking-the-apple-m1-max And that's with the 3090 set at a very high 400W power limit, can get far more efficient when clocked lower. (which is normal, because no dedicated matrix math accelerators on the GPU notably) 2) We'll see, hopefully Apple thinks that the market is worth bothering with…

What do you mean by: "if you can stand Metal for your use case?" What is Metal?

Post by almostdigital » Thu, May 19, 2022, 3:15 AM UTC

    $ conda install pytorch torchvision torchaudio -c pytorch-nightly
    Collecting package metadata (current_repodata.json): done
    Solving environment: failed with initial frozen solve. Retrying with flexible solve.
    Collecting package metadata (repodata.json): done
    Solving environment: failed with initial frozen solve. Retrying with flexible solve.

    PackagesNotFoundError: The following packages are not available from current channels:

      - torchaudio

    OSError: dlopen(/opt/homebrew/Caskroom/miniforge/base/envs/test123/lib/python3.10/site-packages/torchaudio/lib/libtorchaudio.so, 0x0006): Symbol not found: __ZN2at14RecordFunctionC1ENS_11RecordScopeEb