An SLM trained on $8 ESP32-S3
github.com
An SLM trained on $8 ESP32-S3
1–10 of 23 posts
Re: An SLM trained on $8 ESP32-S3
#2This small Klingon speaking language model was trained completely on ESP32. The training took 2 days
Number of parameters: 319K
(Disclaimer) The models is tiny and is not a pocket chatbot. It mistakes and is not capable to support conversation, but that's not the goal of the project
The goals of the project is to bring training to edge devices and it worked out
How can one use it? By using solar panels such device could be turned into autonomous meteorological station
Re: An SLM trained on $8 ESP32-S3
#3Very cool project! Sounds like it was a fun challenge :)
I wonder when we'll start seeing clusters of ESP32-S3s... not sure how interconnects would go though, but I guess the interconnect wouldn't be the bottleneck anyway.
Re: An SLM trained on $8 ESP32-S3
#4hIngan motlh puS ruq tuq DujDaq SISwI' nge'vI'
vIn SuvwI' yIvwI' qarghtaHvIS SIchoH
Re: An SLM trained on $8 ESP32-S3
#5Very cool project! Sounds like it was a fun challenge :) I wonder when we'll start seeing clusters of ESP32-S3s... not sure how interconnects would go though, but I guess the interconnect wouldn't be the bottleneck anyway.
Imagine a Beowulf cluster of those
Re: An SLM trained on $8 ESP32-S3
#6Would probably be better to demonstrate by exsmple how this approach is used to train on sensor data and then use it (as is hinted by the author) instead of acknowledging that the klingon poc is useless.
Re: An SLM trained on $8 ESP32-S3
#7Very cool project.
> Backpropagation (gradients derived by hand)
What does by hand mean in this context?
Also how did you write the readme? It's a curious blend of human and AI writing.
Re: An SLM trained on $8 ESP32-S3
#8Very cool project. > Backpropagation (gradients derived by hand) What does by hand mean in this context? Also how did you write the readme? It's a curious blend of human and AI writing.
The fact that by hand is emphasized so often and so often (see src/handgpt.h comments as well above backward()) makes me think it's AI.
Anyways, it looks like gradients derived by hand means they didn't use autograd. They have written out the expression for dL/dW themselves.
Re: An SLM trained on $8 ESP32-S3
#9Very cool project! Sounds like it was a fun challenge :) I wonder when we'll start seeing clusters of ESP32-S3s... not sure how interconnects would go though, but I guess the interconnect wouldn't be the bottleneck anyway.
Afaik there are already a few videos on YouTube.
Re: An SLM trained on $8 ESP32-S3
#10Very cool project. > Backpropagation (gradients derived by hand) What does by hand mean in this context? Also how did you write the readme? It's a curious blend of human and AI writing.
The fact that by hand is emphasized so often and so often (see src/handgpt.h comments as well above backward()) makes me think it's AI. Anyways, it looks like gradients derived by hand means they didn't use autograd. They have written out the expression for dL/dW themselves.
I think the readme is clearly LLM generated. The tone, the short sentences, the em dashes, lists, sections.. It all feels like LLM.