Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

github.com

Post reply on HN
Print view View original

11–20 of 159 posts

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#11

Post by glimshe »

voidnullvalue wrote: post #3

But why though? Cannot possibly be useful at such slow speeds, and costs a ton to perform that badly

It's not useful for actual work, but the fact it can be run at all shows that we're evolving towards enabling powerful LLMs to run locally.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#12

Post by woadwarrior01 »

sampullman wrote: post #8
Argonautlabs wrote: post #2

Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their release…

This is difficult to read, maybe just link to a gist?

That's because it's copy pasted from a coding agent.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#13

Post by roadside_picnic »

voidnullvalue wrote: post #3

But why though? Cannot possibly be useful at such slow speeds, and costs a ton to perform that badly

I've never understood why "Hacker" News so frequently gets "But why though?" comments at the top.

The entire history of innovation is filled with people doing something just to see they can get it to work, even if badly, and then people continue to iterate on that until it works better, then works well, and then is so obvious people would never even question it. But it all starts with someone doing it to scratch an itch.

Neural networks, the foundation of our current AI revolution, used to fit well into the "neat, but practically useless" category.

Sure there are countless "but why though?" experiments that don't pan out, but that's just the cost of exploration. There can be no step-function innovation in a world where people only do things that make immediate practical sense.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#15

Post by anamexis »

woadwarrior01 wrote: post #12
sampullman wrote: post #8

Earlier quoted context omitted.

This is difficult to read, maybe just link to a gist?

That's because it's copy pasted from a coding agent.

It's difficult to read because it doesn't have line breaks.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#17

Post by bel8 »

pavlov wrote: post #10
Argonautlabs wrote: post #2

Author here. Some context and the caveats up front. The model is Kimi K3, 2.78T parameters, ~1.45 TB of expert weights. It does not fit in memory, so the experts stream from disk: one 17.5 MB file per (layer, expert), read with pread + F_NOCACHE, 16 of 896 per layer. The machine is an M5 Max MacBook Pro with 128 GB and three Thunderbolt 5 enclosures plus the internal SSD. Expert weights are untouched at their release…

This response is so dense with numbers and special characters that it's probably about 1000 tokens. So at 1 token/s, it takes almost 17 minutes to generate this on the MacBook Pro.

And it probably takes longer for a human to compile all that info.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#18

Post by sampullman »

woadwarrior01 wrote: post #12
sampullman wrote: post #8

Earlier quoted context omitted.

This is difficult to read, maybe just link to a gist?

That's because it's copy pasted from a coding agent.

It looks at least partially hand edited to me, although it's getting pretty difficult to tell with Astra...
Post reply on HN