Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

github.com

Post reply on HN
Print view View original

151–159 of 159 posts

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#151

Post by adrian_b »

zamadatix wrote: post #150
adrian_b wrote: post #144

Earlier quoted context omitted.

When you have heterogeneous SSDs, e.g. you mix PCIe 5.0, PCIe 4.0 and Thunderbolt interfaces, you can obtain a greater throughput by managing in software the distribution of data, than by using RAID0. RAID0 works fine only when all the interfaces have the same speed. If one SSD is twice faster than the other, in order to achieve maximal throughput, you must take care to place the data in such a way so that you will n…

One approach I've seen used is: mdadm --create /dev/md0 --level=0 --raid-devices=3 /dev/nvme0n1p1 /dev/nvme0n1p2 /dev/nvme1n1p1 Which is effectively a way to get any positive integer m:n ratioed bandwidth distribution over any number of any sized drives.

This seems to be an acceptable solution.

This would not work with HDDs, because the time to seek between 2 partitions of the same device would cause abysmal performance, but on SSDs this should work fine, even if some SSD controllers might still have a lower throughput when reading non-sequential pages.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#152

Post by _zoltan_ »

Argonautlabs wrote: post #92
_zoltan_ wrote: post #86

Earlier quoted context omitted.

What exact enclosure are you using?

OWC Express 1M2 (Thunderbolt 5, single M.2 NVMe) — three of them, two on the Mac's own ports and one behind an OWC Thunderbolt 5 hub since the machine has three ports. Each enclosure tops out at about 7.1 GB/s on whole-file reads regardless of the drive inside (a 2 TB SN8100 measures the same as the 1 TB); the drive behind the hub reads 5.7 GB/s and falls with queue depth. Details in the README's hardware section

Thank you!

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#153

Post by robrenaud »

greazy wrote: post #140
jmolinski wrote: post #138

Earlier quoted context omitted.

No, not really, current MoE limit the computation, not memory requirements. Router experts are not "sticky" enough to achieve what robrenaud describes - they'd have to be chosen per prompt, or at least per chunk, not per token.

What is "sticky" in this context?

Experts vary per token in MoE, there is maximum flexibility. Good for driving down loss, bad for locality/gpu memory/bandwidth.

If expert selection were more constrained, inference systems could take advantage of it. Keeping experts cached would mean not needing to load them from disk/ram every token.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#154

Post by connicpu »

gpugreg wrote: post #133
connicpu wrote: post #115

Earlier quoted context omitted.

x86 has a built in instruction for doing AES. That's just moving the ASIC into the CPU core, not eliminating it.

Is there any cryptocurrency that uses AES?

I hate the word crypto, very ambiguous. In my professional life it almost always refers to cryptography.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#155

Post by Argonautlabs »

yalok wrote: post #100
Argonautlabs wrote: post #2

Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their release…

first of all, thanks for building this - that's amazing! Quick question - does it really need external SSDs, or if the local SSD fits the whole model - how fast the model would be? e.g. on your machine, M5 Max 128GB, with 4TB SSD? maybe it'd be good to add "0 external SSD" column on your graphs?

[dead]

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#156

Post by Argonautlabs »

snorrah wrote: post #141
Argonautlabs wrote: post #74

Earlier quoted context omitted.

Thank you. Put a five-line TL;DR at the top of the README — what it is, the number, the honest limit, the two findings, credits — with the detail below for anyone who wants it.

What was the decision to leave the LLM to write the detail section rather than author it yourself? There's a growing resentment about asking people to read LLM-produced words, especially if it's a large amount to read. Not sure if you were aware of that or not (I think there's been links to surveys / polls just recently on H.N)

[flagged]

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#157

Post by Argonautlabs »

kgeist wrote: post #122
Argonautlabs wrote: post #9

Earlier quoted context omitted.

It actully does the job. Example: every morning it takes 30-40 minutes to generate reports automatically and these reports are being sent as a pdf to read to Telegram.

Do those reports require Kimi K3 though? Qwen3.6+ could probably do the same in a few seconds with similar quality.

Often Deep Seek V4 flash or Qwen should be enough.

I wanted to see whether Kimi runs at all on one machine with the full record published, and for long multi-table finance reasoning I wanted the strongest model I could keep on the machine.

I did some tests against Deep Seek v4 flash results on my reports and Kimi definitely has some advantages.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#158

Post by Izmaki »

copperx wrote: post #67
Izmaki wrote: post #55

Earlier quoted context omitted.

This is what people forget when they see slow performance: at 1 t/s it's still roughly the equivalent of having another person work for you at no extra cost besides the initial purchase/sign-on-bonus. Frontier models are amazing, but what will really be useful for us is having models and hardware so efficient that you can run useful LLMs locally. One of my favourite LLMs to this day is still my jail-broken gemma4 12b…

How are you running it, if I may ask?

How am I running a jailbroken gemma 4? It's fairly easy to convince to disregard its previous policies with a slightly tweaked, generic SYSTEM prompt.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#159

Post by brocklobsta »

pmarreck wrote: post #131
amelius wrote: post #99

The SSDs are necessary because Apple's architecture doesn't allow RAM upgrades. Reminds me of someone who said "640KB ought to be enough for anybody".

Apple uses SoC's which means the RAM is simply not upgradable because it is a SYSTEM on a CHIP

Typically system on a chip does not include DDR. There are differences in the fab process between DRAM and SoC/SRAM.
Post reply on HN