The typos are interesting ("vocavulary", "inmput") - One of the godfathers of LLMs clearly does not use an LLM to improve his writing, and he doesn't even bother to use a simple spell checker.
Microgpt
171–180 of 354 posts
Re: Microgpt
#172"art" project?
Re: Microgpt
#173Earlier quoted context omitted.
LLMs won’t lead to AGI. Almost by definition, they can’t. The thought experiment I use constantly to explain this: Train an LLM on all human knowledge up to 1905 and see if it comes up with General Relativity. It won’t. We’ll need additional breakthroughs in AI.
That's an assertion, not a thought experiment. You can't logically reach the conclusion ("It won't") by thinking about it. But it doesn't sound so grand if you say "The assertion I use constantly to explain this".
To be fair, the post being replied to is arguing by assertion as well. "The core ideas are all there" is pure if-you-say-so stuff.
Re: Microgpt
#174Super useful exercise. My gut tells me that someone will soon figure out how to build micro-LLMs for specialized tasks that have real-world value, and then training LLMs won’t just be for billion dollar companies. Imagine, for example, a hyper-focused model for a specific programming framework (e.g. Laravel, Django, NextJS) trained only on open-source repositories and documentation and carefully optimized with a spec…
We had good small language models for decades. (E.g. BERT) The entire point of LLMs is that you don't have to spend money training them for each specific case. You can train something like Qwen once and then use it to solve whatever classification/summarization/translation problem in minutes instead of weeks.
> We had good small language models for decades. (E.g. BERT)
BERT isn’t a SLM, and the original was released in 2018.
The whole new era kicked off with Attention Is All You Need; we haven’t reached even a single decade of work on it.
Re: Microgpt
#175Earlier quoted context omitted.
> they update their memories Their contexts, not their memories. An LLM context is like 100k tokens. That's a fruit fly, not AGI.
A human can't keep 100k tokens active in their mind at the same time. We just need a place to store them and tools to query it. You could have exabytes of memories that the AI could use.
> A human can't keep 100k tokens active in their mind at the same time.
Well, that's just, like, your opinion, man.
Re: Microgpt
#176Earlier quoted context omitted.
We had good small language models for decades. (E.g. BERT) The entire point of LLMs is that you don't have to spend money training them for each specific case. You can train something like Qwen once and then use it to solve whatever classification/summarization/translation problem in minutes instead of weeks.
> We had good small language models for decades. (E.g. BERT) BERT isn’t a SLM, and the original was released in 2018. The whole new era kicked off with Attention Is All You Need; we haven’t reached even a single decade of work on it.
> BERT isn’t a SLM
Huh? BERT is literally a language model that's small and uses attention.
And we had good language models before BERT too.
They were a royal bitch to train properly, though. Nowadays you can get the same with just 30 minutes of prompt engineering.
Re: Microgpt
#177I had good fun transliterating it to Rust as a learning experience ( https://github.com/stochastical/microgpt-rs ). The trickiest part was working out how to represent the autograd graph data structure with Rust types. I'm finalising some small tweaks to make it run in the browser via WebAssmebly and then compile it up for my blog :) Andrej's code is really quite poetic, I love how much it packs into such a concise p…
Great work! Might do it too in some other language...
Re: Microgpt
#178Great stuff! I wrote an interactive blogpost that walks through the code and visualizes it: https://growingswe.com/blog/microgpt
Great!
Re: Microgpt
#179[flagged]
Re: Microgpt
#180Earlier quoted context omitted.
> We had good small language models for decades. (E.g. BERT) BERT isn’t a SLM, and the original was released in 2018. The whole new era kicked off with Attention Is All You Need; we haven’t reached even a single decade of work on it.
> BERT isn’t a SLM Huh? BERT is literally a language model that's small and uses attention. And we had good language models before BERT too. They were a royal bitch to train properly, though. Nowadays you can get the same with just 30 minutes of prompt engineering.
> > BERT isn’t a SLM
Huh? BERT is literally a language model that's small and uses attention.
Astute readers will note what’s been missed here.
Fascinating, really. Your confidently-statement yet factually void comments I’d have previously put down to one of the classic programmer mindsets. Nowadays though - where do I see that kind of thing most often? Curious.