The SSDs are necessary because Apple's architecture doesn't allow RAM upgrades. Reminds me of someone who said "640KB ought to be enough for anybody".
Gates never said that, for what it's worth.
Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
101–110 of 156 posts
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#102Earlier quoted context omitted.
If it's (nearly) all streamed from disk can you use an intel/amd machine and a big fat gpu and use the SSDs (nvme for faster throughput) as your virtual vram? Or something like that?
In principle yes, and the upstream engine already has a CUDA path with expert streaming and residency (that's theirs, not ours — we only measured on this Mac). Two things carry over: the experts are read from disk per token either way, and the barrier model — a layer waits for the slowest of its 16 reads — is platform-independent. Two things don't: the 50 GB resident trunk lives in unified memory here, so on a discre…
I eagerly await your update with the benchmarks on this :)
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#103The SSDs are necessary because Apple's architecture doesn't allow RAM upgrades. Reminds me of someone who said "640KB ought to be enough for anybody".
To be fair, nobody has upgradable memory in any system that has enough memory bandwidth and compute power to run LLMs with decent performance. It might be interesting to compare against some decade-old x86 server or workstation stuffed full of LRDIMMs to reach 1.5–2TB of RAM, but the bandwidth would be only slightly faster than a desktop today with high-end DDR5: nowhere close to GPU bandwidth. So performance would still suck.
Designing for extreme expandability comes with pretty steep tradeoffs.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#104We’re reaching quadratic slop. Slop projects that don’t understand what they’re shipping built on top of slop projects that also don’t understand what they’re shipping. Magnificent.
[dead]
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#105The SSDs are necessary because Apple's architecture doesn't allow RAM upgrades. Reminds me of someone who said "640KB ought to be enough for anybody".
You would have the same issue with a GPU. I have 8 slots on my CPU but it cant handle 2.8TB of ram. I cant even buy 2TB for it because even though the mobo is rated for it, nobody ever made those chips (someone will sell me 8x256gb 8Rx4 for $15k but there’s no way the memory controller can handle that). 1TB is $10k and forced it to run slower.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#106The SSDs are necessary because Apple's architecture doesn't allow RAM upgrades. Reminds me of someone who said "640KB ought to be enough for anybody".
Gates never said that, for what it's worth.
> In an April 1985 InfoWorld editorial, James Fawcette wrote that Gates had said something like: “When we set the upper limit of PC-DOS at 640K, we thought nobody would ever need that much memory.”
So yes, Bill Gates denies that story. So it just depends on who you believe.
But to flat out say he never said it is too confident.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#107The SSDs are necessary because Apple's architecture doesn't allow RAM upgrades. Reminds me of someone who said "640KB ought to be enough for anybody".
To be fair, nobody has upgradable memory in any system that has enough memory bandwidth and compute power to run LLMs with decent performance. It might be interesting to compare against some decade-old x86 server or workstation stuffed full of LRDIMMs to reach 1.5–2TB of RAM, but the bandwidth would be only slightly faster than a desktop today with high-end DDR5: nowhere close to GPU bandwidth. So performance would s…
Take a look at AMD’s 12-channel memory servers. The newer Epycs are up to 16-channels now, 1.6TB/s. Pretty great for inference.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#108make it 4x40 raid-0 ssds to achieve 40 tps. or 40 macbooks with each 4 ssd. to get 40 tps.
Bandwidth doesn't multiply like that here, and we measured it rather than assumed it. A MoE layer needs 16 expert reads and can't proceed until the slowest one lands, so a layer costs the max over its reads, not the sum. Going from one drive to four (13.6 → ~33 GB/s of combined ceilings) took decode from ~52% to 100% of our number — not 4× — with Every drive already at 90–100% of its own ceiling. RAID-0 was one of th…
Would the 40 Mac’s work with pipelining though?
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#109Reminds me of Deep Thought from Hitchhiker's Guide to the Galaxy
It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably
running my own locally. I just set the tasks to start when systems go idle over x. Read and copy only to external drive projects, codes, ect for review. I review the reports the changes and apply them myself or correct them. Is it slower than say throwing it into fable yes. But I don't have to be monitoring it 24/7
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#110Earlier quoted context omitted.
Like how crypto used ASICS but then didn't because the scaling of consumer hardware made it obsolete?
To the extent that cryptocurrency moved off ASICs, it was because of interest shifting to different cryptocurrencies that were specifically designed to be harder to mine on an ASIC than Bitcoin's compute-heavy, memory-light hashing. I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to des…
Yet it doesn't seem that ASICs will have any particular advantage over consumer hardware since AI is very memory heavy, which is (right now) expensive no matter how you package it. And the compute is just simple matrix multiplication, which is almost entirely what GPUs were meant to do anyway.