Great article. For a moment, I thought this would be about a gen AI that would turn any input into a “kawai” version of it
Anyway, excellent insights and detail
Towards the Cutest Neural Network
21–26 of 26 posts
Re: Towards the Cutest Neural Network
#22All of this is absurdly complicated. Exactly what I would expect from a new student who doesn't know what they're doing and has no one to teach them how do you engineering in a systematic manner. I don't mean this as an insult. I teach this stuff and have seen it hundreds of times. You should look for "post training static quantization" also called . There are countless ways to quantize. This will quantize both the w…
I kind of see your point, but only in the context of working on time-sensitive task which others rely upon. But if it is hobby/educational project, what is wrong doing things by yourself? And resort to decomposing existing solution if you can't figure out why yours is not working? There's nothing better for understanding something rather than trying to do that "something" from scratch yourself.
I think the point is that OP is learning things about a wide variety of topics that aren't really relevant to their stated goal, i.e. solving the sensor/state inference problem.
Which, as you say, can be valuable! There's nothing wrong with that. But the more complexity you add the less likely you are to actually solve the problem (all else being equal, some problems are just inherently complex).
Re: Towards the Cutest Neural Network
#23My suggestion would be that, since you want a tiny integer-only NN tailored for a specific computer, are only occasionally training one for a specific task, and you have a simulator to generate unlimited data, you simply do random search or an evolutionary method like CMA-ES.
They are easy to understand and code up by hand in a few lines (which is one reason you won't find any libraries for them - they are the 'leftpad' or 'isEven' of NNs, the effort it would take to install and understand and use a library often exceeds what it would take to just write it yourself), will handle any NN topology or numeric type you can invent, and will train very fast in this scenario.
Re: Towards the Cutest Neural Network
#24> since our input data comes from multiple sensors and the the output pose has six components (three spatial positions and three spatial rotations) Typo: two "the" For robotics/inverse pose applications, don't people usually use a 3x3 matrix (three rotations, three spatial) for coordinate representation? Otherwise you get weird gimbal lock issues (I think).
For my application I need just the translations and Euler angles. The range of poses is mechanically constrained so I don't have to worry about gimbal lock. But yeah, my limited understanding matches yours that other parameterizations are more useful in general contexts. This post and interactive explanations have been on my backlog to read and internalize: https://thenumb.at/Exponential-Rotations/ (Also: Thanks for…
Hey there op. I don't know what your sensors are measuring (distance to a point maybe? Or angle from a Valve lighthouse for inside-out tracking?)
But here's my "why didn't you just"
Since you have a forward simulation function (pose to measurements), why didn't you use an iterative solver to reverse it? Coordinate descent is easy to code and if you have a constrained range of poses you can probably just use multiple starting points to avoid getting stuck with a local minimum. Then use the last solution as a starting point for the next one to save iterations.
Sure it's not closed-form like an NN and it can still have pathological cases, but the code is a little more transparent
Re: Towards the Cutest Neural Network
#25Earlier quoted context omitted.
For my application I need just the translations and Euler angles. The range of poses is mechanically constrained so I don't have to worry about gimbal lock. But yeah, my limited understanding matches yours that other parameterizations are more useful in general contexts. This post and interactive explanations have been on my backlog to read and internalize: https://thenumb.at/Exponential-Rotations/ (Also: Thanks for…
Hey there op. I don't know what your sensors are measuring (distance to a point maybe? Or angle from a Valve lighthouse for inside-out tracking?) But here's my "why didn't you just" Since you have a forward simulation function (pose to measurements), why didn't you use an iterative solver to reverse it? Coordinate descent is easy to code and if you have a constrained range of poses you can probably just use multiple…
That's a reasonable idea, but unfortunately wouldn't work in my case since the simulation relies on a lot of scientific libraries in Python and I need the inversion to happen on the microcontroller.
When you say "coordinate descent" do you mean gradient descent? I.e., updating a potential pose using the gradient of a loss term (e.g., (predicted sensor reading - actual sensor reading)**2)?
I bet that would work, but a tricky part would be calculating gradients. I'm not sure if the Python libraries I'm using support that. My understanding is that automatic differentiation through libraries might be easier in a language like Julia where dual numbers flow through everything via the multiple dispatch mechanism.
Re: Towards the Cutest Neural Network
#26Earlier quoted context omitted.
Hey there op. I don't know what your sensors are measuring (distance to a point maybe? Or angle from a Valve lighthouse for inside-out tracking?) But here's my "why didn't you just" Since you have a forward simulation function (pose to measurements), why didn't you use an iterative solver to reverse it? Coordinate descent is easy to code and if you have a constrained range of poses you can probably just use multiple…
That's a reasonable idea, but unfortunately wouldn't work in my case since the simulation relies on a lot of scientific libraries in Python and I need the inversion to happen on the microcontroller. When you say "coordinate descent" do you mean gradient descent? I.e., updating a potential pose using the gradient of a loss term (e.g., (predicted sensor reading - actual sensor reading)**2)? I bet that would work, but a…
Ah makes sense.
No, coordinate descent is a stupider gradient-optional method: https://en.wikipedia.org/wiki/Coordinate_descent
It's slow and sub-optimal, but the code is very easy to follow and you don't have to wonder whether your gradient is correct.