111–120 of 162 posts
We’re reaching quadratic slop. Slop projects that don’t understand what they’re shipping built on top of slop projects that also don’t understand what they’re shipping. Magnificent.
Earlier quoted context omitted.
To the extent that cryptocurrency moved off ASICs, it was because of interest shifting to different cryptocurrencies that were specifically designed to be harder to mine on an ASIC than Bitcoin's compute-heavy, memory-light hashing. I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to des…
Yet it doesn't seem that ASICs will have any particular advantage over consumer hardware since AI is very memory heavy, which is (right now) expensive no matter how you package it. And the compute is just simple matrix multiplication, which is almost entirely what GPUs were meant to do anyway.
Earlier quoted context omitted.
The future of inference is likely in ASICs, so we'll get the inverse, a bit less capable than frontier but super fast models. Like this 14k tok/s beast https://chatjimmy.ai/ from Taalas (who got acquired by AMD recently). GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a…
Like how crypto used ASICS but then didn't because the scaling of consumer hardware made it obsolete?
Earlier quoted context omitted.
To the extent that cryptocurrency moved off ASICs, it was because of interest shifting to different cryptocurrencies that were specifically designed to be harder to mine on an ASIC than Bitcoin's compute-heavy, memory-light hashing. I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to des…
Yet it doesn't seem that ASICs will have any particular advantage over consumer hardware since AI is very memory heavy, which is (right now) expensive no matter how you package it. And the compute is just simple matrix multiplication, which is almost entirely what GPUs were meant to do anyway.
Earlier quoted context omitted.
The future of inference is likely in ASICs, so we'll get the inverse, a bit less capable than frontier but super fast models. Like this 14k tok/s beast https://chatjimmy.ai/ from Taalas (who got acquired by AMD recently). GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a…
Like how crypto used ASICS but then didn't because the scaling of consumer hardware made it obsolete?
Reminds me of Deep Thought from Hitchhiker's Guide to the Galaxy
It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably
Earlier quoted context omitted.
To the extent that cryptocurrency moved off ASICs, it was because of interest shifting to different cryptocurrencies that were specifically designed to be harder to mine on an ASIC than Bitcoin's compute-heavy, memory-light hashing. I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to des…
Yet it doesn't seem that ASICs will have any particular advantage over consumer hardware since AI is very memory heavy, which is (right now) expensive no matter how you package it. And the compute is just simple matrix multiplication, which is almost entirely what GPUs were meant to do anyway.
Earlier quoted context omitted.
So if this all streamed from the internal NVME would you expect better performance?
Just internal 2Tb Macbook M5 Max drive it came in at roughly half the four-drive speed (0.535 vs 1.038 tok/s at 128 tokens), since one fast drive still has to serve all 16 reads per layer while four drives split the load https://raw.githubusercontent.com/argonautlabsai/deltafin/ma...
Earlier quoted context omitted.
Gates never said that, for what it's worth.
> In an April 1985 InfoWorld editorial, James Fawcette wrote that Gates had said something like: “When we set the upper limit of PC-DOS at 640K, we thought nobody would ever need that much memory.” So yes, Bill Gates denies that story. So it just depends on who you believe. But to flat out say he never said it is too confident.
Earlier quoted context omitted.
If it's (nearly) all streamed from disk can you use an intel/amd machine and a big fat gpu and use the SSDs (nvme for faster throughput) as your virtual vram? Or something like that?
In principle yes, and the upstream engine already has a CUDA path with expert streaming and residency (that's theirs, not ours — we only measured on this Mac). Two things carry over: the experts are read from disk per token either way, and the barrier model — a layer waits for the slowest of its 16 reads — is platform-independent. Two things don't: the 50 GB resident trunk lives in unified memory here, so on a discre…