191–200 of 227 posts
It’s all moonspeak to me. I tried reading other comments that explain this and they all sounded different or contradictory. I’ve studied ML as a hobby years ago but this was before the LLM explosion. Guess I need to start over again?
Earlier quoted context omitted.
LLMs are orders of magnitude simpler than brains, and we literally designed them from scratch. Also, we have full control over their operation and we can trace every signal. Are you surprised we understand them better than brains?
"Designed" is a bit strong. We "literally" couldn't design programs to do the interesting things LLMs can do. So we gave a giant for loop a bunch of data and a bunch of parameterized math functions and just kept updating the parameters until we got something we liked.... even on the architecture (ie, what math functions) people are just trying stuff and seeing if it works.
Earlier quoted context omitted.
LLMs are orders of magnitude simpler than brains, and we literally designed them from scratch. Also, we have full control over their operation and we can trace every signal. Are you surprised we understand them better than brains?
We've been studying brains a lot longer. LLMs are grown, not built. The part that is designed are the low-level architecture - but what it builds from that is incomprehensible and unplanned.
LLMs draw origins from, both n-gram language models (ca. 1990s) and neural networks and deep learning (ca. 2000). So we've only had really good ones maybe 6-8 years or so, but the roots of the study go back 30 years at least.
Psychiatry, psychology, and neurology on the other hand, are really only roughly 150 years old. Before that, there wasn't enough information about the human body to be able to study it, let alone the resources or biochemical knowledge necessary to be able to understand it or do much of anything with it.
So, sure, we've studied it longer. But only 5 times longer. And, I mean, we've studied language, geometry, and reasoning for literally thousands of years. Markov chains are like 120 years old, so older than computer science, and you need those to make an LLM.
And if you think we went down some dead-end directions with language models in the last 30 years, boy, have I got some bad news for you about how badly we botched psychiatry, psychology, and neurology!
Earlier quoted context omitted.
I mean... You could? AI comes in all kinds of forms. It's been around practically since Eliza. What is (not) here to stay are the techbros who think every problem can be solved with LLMs. I imagine that once the bubble bursts and the LLM hype is gone, AI will go back to exactly what it was before ChatGPT came along. After all, IMO it's quite true that the AIs nobody talks about are the AIs that are actually doing goo…
Saying that LLMs will disappear once the financial hype desinflate is like saying that LLMs are the answer to everything.
> What is (not) here to stay are the techbros who think every problem can be solved with LLMs.
LLMs are in all likelyhood here to stay, but the scumbags doing business around them right now are hopefully going away eventually.
Earlier quoted context omitted.
Saying that LLMs will disappear once the financial hype desinflate is like saying that LLMs are the answer to everything.
Personally I read the GP post with more emphasis on this bit: > What is (not) here to stay are the techbros who think every problem can be solved with LLMs. LLMs are in all likelyhood here to stay, but the scumbags doing business around them right now are hopefully going away eventually.
After TurboQuant and Gemma 4, came across the following video[0] running Gemma on local machine at 50 token/second. That already looks like Sonnet 3x and 4 level capabilities to me where the model in question (Gemma 4) set ups whole python project with a UI and installs python libraries using uv etc. Add this Simple Self Distillation to the picture and by 2028 I see cheaper coding model providers with much more gener…
I always wonder how much smaller and faster models could be if they were only trained on the latest versions of the languages I use, so for me that is PHP, SQL, HTML, JS, CSS, Dutch, English, plus tool use for my OS of choice (MacOS). Right now it feels like hammering a house onto a nail instead of the other way around.
I’d like to understand AI research better and I recall some posts a while back where someone collected all the key papers that one should read, but I don’t remember enough to be able to find it. Does anyone know what I’m talking about and could link me to that post?
Earlier quoted context omitted.
I've always thought that it is kinda weird that we spend exactly the same amount of compute to calculate both "fork" tokens and "lock" tokens. I think that with grammar-aware sampling / constrained decoding [0][1] it is possible to sometimes skip calling the model altogether if only one token is allowed by grammar and just insert it, but I don't think that any of the current, widely used combinations of models/harnes…
Give coding agents access to intellisense and syntax highlighting. Making coding agents spit out syntactically correct code token by token is like asking a human to code on a whiteboard.
Maybe I'll try that myself, because it feels like it could be a great source of improvements. It would be really useful to see adaptive per-token sampling as an additional decode-only baseline.
"SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6" I know virtually nothing about this area but my naive take is that something that means it still only passes tests around half the time doesn't seem like a particularly big jump forwards. What am I missing?
When designing a benchmark, a pass rate of roughly 50% is useful because it gives you the most information about the relative performance of different models. If the pass rate is 90%+ too often, that means the test is too easy: you're wasting questions asking the model to do things we already know it can do, and getting no extra information. And if it's too low then you're wasting questions at the other end, trying to make it do impossible tasks.