Viewing profile — spmurrayzzz

The biggest gain you'll get is faster memory, provided you have enough capacity to load all the weight into vram. The DGX sparks and Apple silicon memory bandwidth (and also memory…

Slop language aside, the abstract insight is directionally correct I think. CPUs were already becoming much more important during training for test-time scaling, but there you were…

Given the current throughput figures on OpenRouter (~180 tk/s), its likely a much smaller param count on the order of something like Luna. I think the better, more timely compariso…

They've open sourced some of the training and inference code: https://github.com/soofi-project Their work is based on the nemotron arch (so far).

When orgs/bencmarks claim 1% deviation, in most cases that means measuring perplexity loss on datasets like wikitext or c4. Even if the loss is calculated via KLD or similar, its n…

I think its good advice to test both on your own evals for sure, but the MoE parameters are already natively FP4 in ds4. Dropping to 2bpw isn't as big of a loss as it seems (and as…

I had a similar reaction to the "I work at an AI company" and finding out it was Dropbox. And I agree with you, they are not in any way an AI company that would be relevant for som…

I'm not entirely up to date with the latest batch, but I've reviewed some of the rollouts in the past and my sense is that the models are surprisingly good at getting correct custo…

Near-term acquihires are certainly a likely bet I think. But given model progress on related benchmarks like kernelbench [1], I do think a set of more commoditized solutions is als…

I used it, but only briefly to evaluate it. It had some overlap with a tool I built myself, was curious if any of the extra features would be useful. Ultimately I found the data mo…

One of my favorite code comments of all time is still in the src: "# TODO: implement a proper validator to compare against ground truth. For now we just check for exact string matc…

I don't think its smoke and mirrors, though I do have plenty of gripes with how the labs market this product landscape generally speaking. The newest biggest model can still matter…

> You can't on the one hand say "customers are beginning to understand they can spend less" and on the other hand suggest that this is good for forecasts of revenue. Sure you can. …

There is a piece of this I agree with. That you do not need to be a deep technical expert to notice that a company is burning cash by overcommitting to capex, or relying on heroic …

There's examples both in his writing and also in his appearances on podcasts, interviews, etc. I'll cherry pick a couple: “When these new models ‘reason,’ they break a user’s input…

He has also consistently demonstrated, at least to me, that he doesn't really understand how inference works from a technical perspective, which weakens much of his core thesis for…

This depends a bit on your cost sensitivity and what model families you want support for, but Baseten and Fireworks have been my goto. Currently Baseten has ~610ms TTFT and ~82 tk/…

First as an aside, remember that this entire thread is about using local compute. What you're alluding to is some fantasy infinite budget where you have limitless commodity compute…

> That’s kind of a moot point. I don't believe it's moot, but I understand your point. The fact that models are memory bandwidth bound does not at all mean that other overhead is i…

No I'm saying there are quite a few more bottlenecks than that (I/O being a big one). Even in the more efficient training frameworks, there's per-op dispatch overhead in python its…

For inference, even with continuous batching, getting 100% MFUs is basically impossible to do in practice. Even the frontier labs struggle with this in highly efficient infiniband …

Bits per weight, its an average precision across all the weights. When you quantize these models, they don't just used a fixed precision size across all model layers/weights. There…

I've tested this myself often (as an aside: I'm in said community, I run 2x RTX Pro 6000 locally, 4x 3090 before that), and I think what you said re: "willing to wait" is probably …