331–340 of 345 posts
This might be silly, but … since the assistant models are so much smaller than the full models. What if we just use those smaller models? Any idea how much worse they will be ? Or is the issue that their error will really diverge as you accept more of their tokens?
Earlier quoted context omitted.
A few days ago I switched again from Qwen3.6 to Gemma 4 - for personal use I've experienced better average performance with the 26B version of the latter than the 27B of the former. For someone who's been running local models for a long while, these are very very exciting times.
Oh that's fascinating. 3.6 27B is pretty damned good, but slow in wall-clock times on my DGX Spark-alike. It generates huge reams of thinking before it gets the (usually correct!) answer, so wall-clock time is rough for tasks even at ~20tk/s I'm surprised the 26B-A4B is better? It should be faster too, interesting. I'm excited to try 31B with MTP, because MTP-2 is what makes 27B bearable on the GB10. What are you usi…
For coding I don't need image support so I stuff the entire GPU with text-only mode. I don't have a workflow where I send LLMs off to generate thousands of lines of code but what little coding I did I did with Qwen3.6 and it was spectacular, as you likely suggest.
Earlier quoted context omitted.
Yeah, sorta. But then I don't see us killing or even just laying off people to meet the climate goals (the point is to save people / well-being, not reduce it), whereas we can choose which electric technologies to use so long as emissions from electricity are dominated by the fossil components, so I don't really see the "could replace humans with more efficient workers" math working out this way
> the point is to save people / well-being, not reduce it Oh, you haven't met _that_ part of the climate people. A surprising number of them do want to reduce the number of people and they see "degrowth" as the solution.
I can see how it appears that way but ultimately that's nobody's goal of course. Might be worth actually talking to someone who you feel is in that group and realizing that they have the same morals and end goals as you and me, just seeing a different path to get there
Many would actually say we should reduce the well-being, if you want to take it literally, but specifically of the richest 10% of people or so, such that everyone can be at an equal lifestyle that earth can sustain, since it's not fair if 90% needs to live far under that common standard so that the rich can be rich. That could be something to agree or disagree with (most of us here are in that top 10%; I certainly am), but I expect you'd not find 99% of "them" having an unreasonable stance when you hear them out
Earlier quoted context omitted.
https://chatjimmy.ai being a demo of the "burn the model to an ASIC" approach being sold by Taalas[0], an approach which they use to run Llama 3.1 8B at ~17000 tokens per second. [0] - https://taalas.com/products/
Not to downplay their accomplishment but Llama 3.1 8B is a terrible model. It's really outdated at this point. It's cool that they were able to accelerate a model with silicon, but it also feels wasteful since llama 8B is such a useless model?
Earlier quoted context omitted.
for anyone wanting a glossary to explain the naming scheme here: E4B = 4B effective parameters (using per-layer embeddings) E2B = 2B (like above) it = instruction tuned (rlhf and all that jazz) assistant = Multi-token drafters (the new 2x speed up)
> assistant naming still hard I see
google/gemma-4-31B-it-ass
Earlier quoted context omitted.
for anyone wanting a glossary to explain the naming scheme here: E4B = 4B effective parameters (using per-layer embeddings) E2B = 2B (like above) it = instruction tuned (rlhf and all that jazz) assistant = Multi-token drafters (the new 2x speed up)
> assistant naming still hard I see
Earlier quoted context omitted.
Qwen is better at some things (code, in particular), but Gemma has better prose and better vision. At least, it feels that way to me.
gemma is also just way faster. i dont wanna wait 10min to get a 5-10% better answer (and sometimes, actually worse answer). best is to use your own model router atm, depending on the task
Earlier quoted context omitted.
Qwen is still better that Gemma though. Also you can tune it more for different tasks, which means that you can prioritize thinking and accuracy versus inference speed.
Genuine question: how do you tune it? I thought "fine-tuning" meant training it on additional data to add additional facts / knowledge? I might be mistaking your use of the word "tune", though :)
Most clients that support ollama support passing extra body options where you can set those.
An LLM forward inference doesn't just predict token vectors for the new last token:
In diagrams the forward pass is typically depicted as taking input tokens vectors (here native context being 100 for didactic purposes) and generating output token vectors .
As far as I understand that is didactically only semi correct, it correctly depicts the locations of tokens in the input and output string, but actually the token vector at the t2 output position is NOT identical to the t2 vector from the input, but a token vector which after softmax gives P(t2 | t1).
And output token position t5 actually corresponds to P(t5 | t1,t2,t3,t4). I.e. the forward inference is modelling the statistical conditional N-gram function from inputs to outputs, from the bigram conditional probability P(t2 | t1) all the way up to P(t101 | t1, t2, t3, ..., t98, t99, t100).
Suppose you want to take bigger steps, nothing prevents one from calculating the forward function by sliding a fixed (committed output string) to the left not 1 position but say 10 positions, and then using the last 10 predictions as the new output prediction. That doesn't need a new MTP model. Perhaps it would take some careful modification to ensure the same original output distributions as if the tokens were generated one at a time, but this hints at the possibility.
One could also slide to the left 5 positions twice, not committing to all 10 new tokens at once but only commiting to the 5 oldest values of the 10 new values, and using the noncommited 5 last values as input vectors for the next invocation, so the model can push the new 5 vectors towards its final commited output vector value in 2 steps for better convergence...
Is there any reason multitoken prediction doesn't work this way, or is there some aspect of the conditional N-gram interpretation of LLM models that I am miscomprehending?
Earlier quoted context omitted.
This is something I've been thinking about for a while...the current state of things really does feel kind of like the dialup era, wondering what the "broadband" era could look like. Watching tokens stream in is reminiscent of watching a jpeg load a few rows of pixels at a time, and the various different loading and connecting animations that applications implemented before things got fast enough to make them less re…
Take a look at https://chatjimmy.ai/ -- it's running against Taalas' "hardcore" silicon model, ie a dedicated, ASIC-like chip.