Earlier quoted context omitted.
I made an explicit reverse pass (no autodiff), it was 8x faster in Python
I made an explicit double-reverse pass (no code!), it was 80x faster in my head!
"I've got an ipod -- In My Mind"
Microgpt
331–340 of 354 posts
Re: Microgpt
#332Sorry to RFELI5 but but ... I thought a "token" was a word? The example is of names and the output is new improvised names, implying that a character is a token? Or do all LLMs operate at character level?
Also is there some minima of training data? E.g. if you just trained on "True" "False" I assume it would be .5 Bernoulli? What is the minimum to see "interesting" results I guess.
Re: Microgpt
#333Someone has modified microgpt to build a tiny GPT that generates Korean first names, and created a web page that visualizes the entire process [1]. Users can interactively explore the microgpt pipeline end to end, from tokenization until inference. [1] English GPT lab: https://ko-microgpt.vercel.app/
This kind of thing is pretty easy to do with a much leaner model https://docs.pytorch.org/tutorials/intermediate/char_rnn_gen...
I assume the goal isn't to generate Korean names but to learn GPTs.
Re: Microgpt
#334Inspiring. Definitely got nerd sniped by this. Now you can train it in under a second on one CPU core with no dependencies: https://github.com/Entrpi/eemicrogpt
Detailed optimizing journey in the readme too.
Re: Microgpt
#335Earlier quoted context omitted.
A human can't keep 100k tokens active in their mind at the same time. We just need a place to store them and tools to query it. You could have exabytes of memories that the AI could use.
> A human can't keep 100k tokens active in their mind at the same time. Well, that's just, like, your opinion, man.
It's hard to know what this means, but... really? I mean most people can't keep more than 10 digits in their mind at a time.
Re: Microgpt
#336Earlier quoted context omitted.
I made an explicit reverse pass (no autodiff), it was 8x faster in Python
Can you share a link?
https://www.ideone.com/VAz4Nn
Doesn't run inside IDEone due to the external download link, but you can copy&paste the code over
Re: Microgpt
#337Earlier quoted context omitted.
> Astute readers will note what’s been missed here. I’m not astute enough to see what was missed here. Could you explain?
If I'm not mistaken, BERT is a classifier (enters text, outputs labels) so it is not a "Language model", as it cannot be used for text generation.
The abstract of the original BERT paper starts with these words: "We introduce a new language representation model called BERT, [...]" The paper itself contains the phrase "language model" 24 times.
It might not be considered a language model today, but it was certainly considered one when it was originally published. Or so it would seem to me. Maybe there is a semantic shift which happened here?
Re: Microgpt
#338I had good fun transliterating it to Rust as a learning experience ( https://github.com/stochastical/microgpt-rs ). The trickiest part was working out how to represent the autograd graph data structure with Rust types. I'm finalising some small tweaks to make it run in the browser via WebAssmebly and then compile it up for my blog :) Andrej's code is really quite poetic, I love how much it packs into such a concise p…
how did you do the transliteration/port?
Re: Microgpt
#339Earlier quoted context omitted.
Great work! Might do it too in some other language...
I got a convertion to Java. It worked (at least I think...) in the first try. Then I want to convert this to my own programming language (which traspiles to C). I like those tiny projects very much!
how did you do the transliteration/port?
Re: Microgpt
#340Earlier quoted context omitted.
I made an explicit reverse pass (no autodiff), it was 8x faster in Python
I made an explicit double-reverse pass (no code!), it was 80x faster in my head!
code here, it's just not interesting to look at: