If people like "give me the simplest possible working coding example" of neural networks, I highly recommend this one:
"A Neural Network in 11 lines of Python (Part 1)": https://iamtrask.github.io/2015/07/12/basic-python-network/
Towards the Cutest Neural Network
11–20 of 26 posts
Re: Towards the Cutest Neural Network
#12All of this is absurdly complicated. Exactly what I would expect from a new student who doesn't know what they're doing and has no one to teach them how do you engineering in a systematic manner. I don't mean this as an insult. I teach this stuff and have seen it hundreds of times. You should look for "post training static quantization" also called . There are countless ways to quantize. This will quantize both the w…
Despite the tone this is excellent advice! I had similar impressions reading the article and was wondering if I missed something.
Re: Towards the Cutest Neural Network
#13Nice write‑up. A couple of notes from doing roughly the same dance on Cortex‑M0 and M3 boards for sensor fusion. 1. You can, in fact, get rid of every FP instruction on M0. The trick is to pre‑bake the scale and zero_point into a single fixed‑point multiplier per layer (the dyadic form you mentioned). The formula is ini Copy Edit y = ((W x + b) M) >> s Where M fits in an int32 and s is the power‑of‑two shift. You com…
Awesome, thanks! This is exactly the kind of experienced take I was hoping my blog post would summon =D
Re: computing M and s, does torch.quantization.quantize_qat do this or do you do it yourself from the (presumably f32) activation scaling that torch finds?
I don't have much experience with this kind of numerical computing, so I have no intuition about how much the "quantization" of selecting M and s might impact the overall performance of the network. I.e., whether
- M and s should be trained as part of QAT (e.g., the "Learned Step Size Quantization" paper)
- it's fine to just deterministically compute M and s from the f32 activation scaling.
Also: Thanks for the tips re: CMSIS-NN, glad to know it's possible to use in a non-framework way. Any chance your example is open source somewhere?
Re: Towards the Cutest Neural Network
#14What benefit does jax.nn provide over rolling one's own? There are countless examples on the web of small neural networks, written from scratch.
Could you point to an example that you like more? One of the author’s goals is to: > solicit “why don’t you just …” emails from experienced practitioners who can point me to the library/tutorial I’ve been missing =D (see the alternatives-considered down the page for what I struck out on)
[deleted]
Re: Towards the Cutest Neural Network
#15All of this is absurdly complicated. Exactly what I would expect from a new student who doesn't know what they're doing and has no one to teach them how do you engineering in a systematic manner. I don't mean this as an insult. I teach this stuff and have seen it hundreds of times. You should look for "post training static quantization" also called . There are countless ways to quantize. This will quantize both the w…
I kind of see your point, but only in the context of working on time-sensitive task which others rely upon. But if it is hobby/educational project, what is wrong doing things by yourself? And resort to decomposing existing solution if you can't figure out why yours is not working?
There's nothing better for understanding something rather than trying to do that "something" from scratch yourself.
Re: Towards the Cutest Neural Network
#16Experienced practitioner here, the second half pf the post describes doing everything exactly the way I have done it (only differences are I picked C++ and Eigen instead of rust and nalgebrafor inference, and i used torch’s ndarray and backprop tools instead of jax’s- with the analagous “just print out a C++ code from python” approach to weight serialization). You picked up on the key insight which is that the size of the code needed to just directly implement the inference equations is much smaller than the size of the configuration file of any possible framework that was flexible enough to meet your requirements of (rust, no inference time allocation, no inference time floating point, trained from scratch, ultra small parameter count, …)
Re: Towards the Cutest Neural Network
#17The last time I did anything like this, the easiest workflow I found was to use your favorite high-level runtime for training and just implement a serializer converting the model into source code for your target embedded system. Hand-code the inference loop. This is exactly the strategy TFA landed on.
One advantage of having it implemented in code is that you can observe and think about the instructions being generated. TFA didn't talk at all about something pretty important for small/fast neural networks -- the normal "cleanup" code (padding, alignment, length alignment, data-dependent horizontal sums, etc) can dwarf the actual mul->add execution times. You might want to, e.g., ensure your dimensions are all multiples of 8. You definitely want to store weights as column-major instead of row-major if the network is written as vec @ mat instead of mat @ vec (and vice versa for the latter).
When you're baking weights and biases into code like that, use an affine representation -- explicitly pad the input with the number one, along with however many extra zeroes you need for any other length padding requirements make sense for your problem (usually zero for embedded, but this is a similar workflow to low-resource networks on traditional computers, where you probably want vectorization).
Floats are a tiny bit hard to avoid for dot products. For similar precision, you require nearly twice the bit count in a fixed-point representation just to make the multiplies work, plus some extra bits proportional to the log2 of the dimension. E.g., if you trained on f16 inputs then you'll have roughly comparable precision with i32 fixed-point weights, and that's assuming you go through the effort to scale and shift everything into an appropriate numerical regime. Twice the instruction count (or thereabouts) on twice the register width makes fixed-point 2-4x slower for similar precision than a hardware float, supposing those wide instructions exist for your microcontroller, and soft floats are closer to 10x slower for multiply-accumulate. If you're emulating wide integer instructions, just use soft floats. If you don't care about a 4x slowdown, just use soft floats.
Training can be a little finicky for small networks. At a minimum, you probably want to create train/test/validate sets and have many training runs. There are other techniques if you want to go down a rabbit hole.
Other ML architectures can be much more performant here. Gradient-boosted trees are already SOTA on many of these problems, and oblivious trees map extremely well to normal microcontroller instruction sets. By skipping the multiplies, your fixed-point precision is on par with floats of similar bit-width, making quantization a breeze.
Re: Towards the Cutest Neural Network
#18This is such a confused blogpost I swear this had to be a teenager just banging their head against the wall.
Wanting to natively train a quantized neural network is stupid unless you are training directly on your microcontroller. I was constantly waiting for the author to explain their special circumstances and it turns out they don't have any. They just have a standard TinyML [0] use case that's been done to death with fixed point quanitization aware training, which unlike what the author of the blog post said, doesn't rely on terabytes of data.
QAT is done on a conventionally trained model with much less data than the full training process. Doing QAT early has no benefits. The big downside of QAT isn't that you need a lot of data, it's that you need the same data distribution as the original training data and nobody has access to that, because only the weights are published.
[0] https://medium.com/@thommaskevin/tinyml-quantization-aware-t...
Re: Towards the Cutest Neural Network
#19Out of curiosity, did you consider bayesian state estimators?
For example, an unscented kalman filter: https://www.mathworks.com/help/control/ug/nonlinear-state-es...
Re: Towards the Cutest Neural Network
#20All of this is absurdly complicated. Exactly what I would expect from a new student who doesn't know what they're doing and has no one to teach them how do you engineering in a systematic manner. I don't mean this as an insult. I teach this stuff and have seen it hundreds of times. You should look for "post training static quantization" also called . There are countless ways to quantize. This will quantize both the w…
Targeting ONNX and using something like https://github.com/kraiskil/onnx2c as parent mentioned is good advice.