111–120 of 246 posts
Didn't expect it to beat 3.8 27B so cleanly. Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.
Earlier quoted context omitted.
If/when we can get larger context this will mostly be mitigated by these smaller models being able to search the internet. Self-learning/improving would be even better but that's still a long way to go.
Search results suck because the web sucks these days. The big models from OpenAI/Anthropic have every book in existence baked into them
Earlier quoted context omitted.
You assume that openai's inference is profitable and that they aren't just trying to bolster revenue before their IPO. The only indication that openai is profitable comes from openai (whom I wouldn't trust with any statement, especially when it comes to profitability). In fact there is evidence that inference is not profitable simply because the rate of losses doesn't seem to reduce as revenue increases: if inference…
I don’t pay OpenAI’s bills - I pay what they charge me. Their cost accounting isn’t relevant to a user.
Earlier quoted context omitted.
I don't like these comparisons. Sure it is impressive, but it does not have a world knowledge of larger models. It has most of theirs intelligence.
In a lot of spaces, this is actually preferable. Ex - nodejs natively supports a huge set of typescript with built-in type stripping these days. But ask most hosted models to build a typescript project and they default to a heavy compile step, or a tool like tsx, ts-node, etc. Models with lots of "world knowledge" have a good chunk of that knowledge go stale, and there's no real way to refresh it without training a n…
In a discussion on economic history, say, someone will opine that Alexander Hamilton had some particular opinion about tariff policy… based on their having a vague memory of a blog post where someone quoted a passage in support of some point. But wait - you can search the federalist papers, the text’s right there to be read, before you commit to saying online ‘Hamilton thought tariffs were a great idea’ you could take your internal ‘I seem to recall reading something about hamilton’s opinion on tariffs’ thought and turn it into a little RAG query where you pull up a source and check before you put another factoid out onto the internet.
And so I feel absolutely the same way about LLMs. I don’t care how much factual information was in the training data, when the LLM wants to rely on something it vaguely recalls having been trained on, it owes it to me to dig up a source and vet it.
There are limits to this, of course. I don’t want it to be thinking ‘but wait, maybe my memory of Python syntax is faulty. Is = used for assignment? …’.
But in general some caution about repeating vaguely recalled easily checked facts is warranted.
Adding to my homelab stack, hopefully it doesn't overthink like the little model. Actually, hoping it thinks a bit less. Wait actually I'm really praying it reasons a bit more directly. But wait, I'm really sure that it must be a bit better.
Same reason your phone has a few big CPU cores for real work, it's much better to "race to idle" than have an "efficient" core struggle. Shitty experience, shitty power efficiency.
I think these "Flash" models are sort of an evolutionary dead end. Sure, there are some routine tasks and applications where they can be used. But for the actual novel development work? It's much better to run a big model at high power for 30 mins than watch the Flash model struggle for 2 hours and produce massive churn. Same reason your phone has a few big CPU cores for real work, it's much better to "race to idle"…
Not to mention, they’re great for self-hosting and getting yourself to not be dependent on some API that can go down or be altered at any time.
Big models seem to mostly be good for pushing ahead the frontier - the smaller models tend to gain the frontier’s capabilities after only a handful of months anyway. Many are perfectly content remaining a few months behind the bleeding edge.
Adding to my homelab stack, hopefully it doesn't overthink like the little model. Actually, hoping it thinks a bit less. Wait actually I'm really praying it reasons a bit more directly. But wait, I'm really sure that it must be a bit better.
You’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each: 1. It overthinks — Just like the previous iteration. High confidence. 2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others. 3. It sometimes overthinks — Best case all around. A feature, not an impairment. One final thing worth mentioning: (I made myself irrationall…
Earlier quoted context omitted.
You’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each: 1. It overthinks — Just like the previous iteration. High confidence. 2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others. 3. It sometimes overthinks — Best case all around. A feature, not an impairment. One final thing worth mentioning: (I made myself irrationall…
> Three honest possibilities, and I’ll be straight with you about each This. I don't know if the "honest answer" phrasing is part of the system prompt or alignment, but when people say "honestly" all the time I start wondering how honest they're being.