Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinygrad: A simple and powerful neural network framework

tinygrad.org

Post reply on HN
Print view View original

131–140 of 147 posts

Re: Tinygrad: A simple and powerful neural network framework

#131

Post by georgehotz »

twothreeone wrote: post #126
georgehotz wrote: post #97

Earlier quoted context omitted.

> they're dynamically shaped and hit near peak While this is true for most common GEMM looking ops, if you tread off the beaten path things get slow (odd channel sizes, batch sizes, etc...). Right now in PyTorch, GroupNorm is 2x slower than BatchNorm. There's no fundamental reason, just that the kernels loop over axes in a less than ideal order. Dynamic recompilation allows you to change the loop order too, not just…

> Right now in PyTorch, GroupNorm is 2x slower than BatchNorm How did you benchmark this? I think there are like 3 or 4 different GN implementations in PyTorch..

Whole net performance at comma, when we switch from BatchNorm to GroupNorm it adds 70ms to the training step time, and it's -70ms for no norm. We also wrote a custom AllNorm that's like 10% slower than BatchNorm (and I put several hours into trying to optimize it). Obviously not indicative of everyone's experience, but my point is BatchNorm is hyperoptimized and others, which are pretty much the same thing, aren't.

Re: Tinygrad: A simple and powerful neural network framework

#132

Post by twothreeone »

georgehotz wrote: post #131
twothreeone wrote: post #126

Earlier quoted context omitted.

> Right now in PyTorch, GroupNorm is 2x slower than BatchNorm How did you benchmark this? I think there are like 3 or 4 different GN implementations in PyTorch..

Whole net performance at comma, when we switch from BatchNorm to GroupNorm it adds 70ms to the training step time, and it's -70ms for no norm. We also wrote a custom AllNorm that's like 10% slower than BatchNorm (and I put several hours into trying to optimize it). Obviously not indicative of everyone's experience, but my point is BatchNorm is hyperoptimized and others, which are pretty much the same thing, aren't.

Thanks, that's certainly helpful anecdotal evidence.. yeah it seems like there should be an "AllNorm" implementation that covers all cases and is just fast. I was wondering because I'm currently looking at math_group_norm, which was ported from PyTorch/XLA and it results in a really weird decomposition that I'm astonished works at all. https://github.com/pytorch/pytorch/blob/master/aten/src/ATen...

I'm also wondering if the handcoded backward passes are actually "numerically correct", because e.g. epsilon doesn't appear in it at all. Someone worked out the gradients manually for BN here: https://web.archive.org/web/20180826123459/http://cthorey.gi...

You can clearly see epsilon appearing in the output. And of course there's the whole training vs. eval mode thing with BN which GN doesn't have.

In any case, thanks again.

Re: Tinygrad: A simple and powerful neural network framework

#133

Post by orf »

KptMarchewa wrote: post #109
jamesrom wrote: post #23

tinygrad core is over 1000 loc now[1]. If anyone was looking for a fun weekend project :) https://github.com/geohot/tinygrad/blob/master/.github/workf...

It does achieve that by being most horizontally dense Python code I've ever seen.

Wow https://github.com/geohot/tinygrad/blob/master/tinygrad/tens...

Re: Tinygrad: A simple and powerful neural network framework

#134

Post by adamsmith143 »

DeathArrow wrote: post #3

I believe neural networks are over hyped sometimes. They are not always the best tool for the job. There are lots of other ML techniques such as SVM, naive Bayes, k-nearest neighbor, decision tree, logistic regression, random forest etc. nobody is using because they lack the hype factor. If something lacks some keywords like neural network, deep learning, reinforced learning, than it is deemed not cool.

Don't think you really know the field. On my team we almost exclusively use XGBoost or other boosted tree methods because it is typically the best model for tabular data. If we were working on CV or NLP that would be a different story and for that Neural Nets are by far the best models.

Re: Tinygrad: A simple and powerful neural network framework

#135

Post by adamsmith143 »

zelphirkalt wrote: post #12
learndeeply wrote: post #6

Earlier quoted context omitted.

I can't think of anything that neural nets can't beat, except small tabular data with boosted decision trees. Can you give some examples?

Explicability is a big part of it It is often worth being a percent less accurat but having an explainable result.

I've been on a lot of ML teams and outside of Finance and a few other sensitive topics explainability has always been irrelevant.

Re: Tinygrad: A simple and powerful neural network framework

#136

Post by adamsmith143 »

DeathArrow wrote: post #10
minimaxir wrote: post #5

Earlier quoted context omitted.

The problems where traditional ML works best and the problems where Transformers or ConvNets work best are usually two different domains. AI is not a buzzword.

>The problems where traditional ML works best and the problems where Transformers or ConvNets work best are usually two different domains. Yes and we are using NN for everything.

But we aren't. Outside of using AEs for embeddings and then feeding them through a boosted tree model I don't know anyone using NNs for tabular data. We all use XGBoost or Catboost, etc.

Re: Tinygrad: A simple and powerful neural network framework

#137

Post by therealchiggs »

There's an interesting roadmap in the "cherry" folder of the git repo[0]. It begins by bringing up a design on FPGA and ends with selling the company for $1B+ by building accelerator cards to compete with NVIDIA:

  Cherry Three (5nm tapeout)
  =====
  * Support DMA over PCI-E 4.0. 32 GB/s
  * 16 cores
  * 8M elements in on board RAM of each core (288 MB SRAM on chip)
  * Shared ~16GB GDDR6 between cores. Something like 512 GB/s
  * 16x 32x32x32 matmul = 32768 mults
  * 1 PFLOP @ 1 ghz (finally, a petaflop chip)
  * Target 300W, power savings from process shrink
  * This card should be on par with a DGX A100 and sell for $2000

  * At this point, we have won.
  * The core Verilog is open source, all the ASIC speed tricks are not.
  * Cherry will dominate the market for years to come, and will be in every cloud.
  * Sell the company for $1B+ to anyone but NVIDIA
[0] https://github.com/geohot/tinygrad/blob/master/accel/cherry/...

Re: Tinygrad: A simple and powerful neural network framework

#138

Post by zelphirkalt »

adamsmith143 wrote: post #135
zelphirkalt wrote: post #12

Earlier quoted context omitted.

Explicability is a big part of it It is often worth being a percent less accurat but having an explainable result.

I've been on a lot of ML teams and outside of Finance and a few other sensitive topics explainability has always been irrelevant.

What happens, when your model exhibits a discriminating bias? How do you find out, what is going wrong? Knowing, what the model pays attention to can be pretty helpful.

Re: Tinygrad: A simple and powerful neural network framework

#139

Post by thrtythreeforty »

brrrrrm wrote: post #121
thrtythreeforty wrote: post #118

Earlier quoted context omitted.

Or, better, identifying that the machine has a primitive that is better than doing each op individually. For example, a multiply-accumulate instruction vs a multiply and separate accumulate. The source code still says "a*b+c", the compiler is just expected to infer the MAC instruction.

Yep! This is an assumed optimization when it comes to modern linear algebra compilers. New primitives go way beyond FMAs: full matrix multiplies on nvidia/Intel and outer product accumulates on Apple silicon. It’s also expected that these are used nearly optimally (or you’ve got a bug).

I am extremely familiar with how far these primitives go, ha. I develop kernels professionally for AWS ML accelerators.

Re: Tinygrad: A simple and powerful neural network framework

#140

Post by nl »

Vetch wrote: post #68
nl wrote: post #57

Earlier quoted context omitted.

> Sentiment analysis trained on the standard datasets is one place where performance is barely better than old-school linear classifiers Well yeah. But why would you do that? Do what eveyrone does: Train on large scale a language corpus (or use a pre-trained model) then finetune for sentiment analysis. > I strongly advice against using something fine-tuned solely on sentiment datasets Did you mean trained on sentimen…

No, I meant finetuned. I also meant finetuned when I said trained. Experience with applying finetuned sentiment classifiers on real world data found gain vs cost of running to not be worth it. They remain nearly as brittle as cheaper classifiers and have a habit of gloming too much unto certain adjectives. They are also prone to overfitting on finetuned data's domain. Transformers trained not specifically on sentimen…

Well when you put the sentiment head on a pretrained language model you kind of have to train that head a bit on the sentiment task right?

But if the rest of your model is frozen the head will never see actual words, just contextual vectors from the LM.

It feels like we are in strong agreement but using slightly different terms or something

Post reply on HN