131–140 of 354 posts
> [p for mat in state_dict.values() for row in mat for p in row] I'm so happy without seeing Python list comprehensions nowadays. I don't know why they couldn't go with something like this: [state_dict.values() for mat for row for p] or in more difficult cases [state_dict.values() for mat to mat*2 for row for p to p/2] I know, I know, different times, but still.
Earlier quoted context omitted.
It can learn. When my agents makes mistake they update their memories and will avoid making the same mistakes in the future. >Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. Tools like Claude Code and Codex have used RL to train the model how to use the harness and make a ton of money.
That's not learning, though. That's just taking new information and stacking it on top of the trained model. And that new information consumes space in the context window. So sure, it can "learn" a limited number of things, but once you wipe context, that new information is gone. You can keep loading that "memory" back in, but before too long you'll have too little context left to do anything useful. That kind of cap…
One of the biggest boosts in LLM utility and knowledge was hooking them up to search engines. Giving them the ability to query a gigantic bank of information already has made them much more useful. The idea that it can't similarly maintain its own set of information is shortsighted in my opinion.
Earlier quoted context omitted.
It can learn. When my agents makes mistake they update their memories and will avoid making the same mistakes in the future. >Reinforcement learning, on the other hand, can do that, on a human timescale. But you can't make money quickly from it. Tools like Claude Code and Codex have used RL to train the model how to use the harness and make a ton of money.
> they update their memories Their contexts, not their memories. An LLM context is like 100k tokens. That's a fruit fly, not AGI.
Earlier quoted context omitted.
what gut? we are already doing that. there are a lot of "tiny" LLMs that are useful: M$ Phi-4, Gemma 3/3n, Qwen 7B... There are even smaller models like Gemma 270M that is fine tuned for function calls. they are not flourish yet because of the simple reason: the frontier models are still improving. currently it is better to use frontier models than training/fine-tuning one by our own because by the time we complete t…
I’ve tried those tiny LLMs and they don’t seem useful to me for real world tasks. They are toys for super simple autocomplete.
Oh yeah:
> The next big tech trend will start out looking like a toy
>Author and investor Chris Dixon explains why the biggest trends start small — and often go overlooked.
> [p for mat in state_dict.values() for row in mat for p in row] I'm so happy without seeing Python list comprehensions nowadays. I don't know why they couldn't go with something like this: [state_dict.values() for mat for row for p] or in more difficult cases [state_dict.values() for mat to mat*2 for row for p to p/2] I know, I know, different times, but still.
[for p in row in mat in state_dict.values()]
I wrote a C++ translation of it: https://github.com/verma7/microgpt/blob/main/microgpt.cc 2x the number of lines of code (~400L), 10x the speed The hard part was figuring out how to represent the Value class in C++ (ended up using shared_ptrs).