Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

github.com

Post reply on HN
Print view View original

61–70 of 159 posts

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#61

Post by Capricorn2481 »

redox99 wrote: post #42
schmorptron wrote: post #38

Earlier quoted context omitted.

It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably

At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.

> At 1t/s it's still faster than humans for a lot of tasks

Which tasks? I think you're underestimating how token hungry current proposed workflows are.

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#62

Post by bluedino »

IgorPartola wrote: post #41
schmorptron wrote: post #38

Earlier quoted context omitted.

It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably

My parents learned to program on punch cards. They told me it was a day of preparing the program, an hour of running it, just to get a syntax error.

Write the program, punch the cards, send the cards to another building to be loaded, program runs, printout comes out in another building, somehow this takes 2-3 days

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#63

Post by redox99 »

Capricorn2481 wrote: post #61
redox99 wrote: post #42

Earlier quoted context omitted.

At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.

> At 1t/s it's still faster than humans for a lot of tasks Which tasks? I think you're underestimating how token hungry current proposed workflows are.

Anything you do right now? A typical 10 minute prompt "simply" becomes about 7 hours long. (40t/s vs 1t/s).

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#64

Post by RugnirViking »

jgalt212 wrote: post #33
dusted wrote: post #31

A medium prompt in only 11 days.

A medium prompt = 1 million tokens?

How many times have you had a model start compacting already before getting back to you? Most have 1 million context window. It's happened to me occasionally

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#65

Post by redox99 »

mhaberl wrote: post #60
redox99 wrote: post #42

Earlier quoted context omitted.

At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.

I would agree, but I want to add that I have real issues with combination of opencode plus slow inference (4-5tok/s). I get weird interruptions. I can only guess its related to some kind of timeouts in the harness or something. Its not a problem of the model of course, but it seems impractical atm. I wonder if anyone else had this kind of thing happening.

I think pi handles it better

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#66

Post by Capricorn2481 »

redox99 wrote: post #63
Capricorn2481 wrote: post #61

Earlier quoted context omitted.

> At 1t/s it's still faster than humans for a lot of tasks Which tasks? I think you're underestimating how token hungry current proposed workflows are.

Anything you do right now? A typical 10 minute prompt "simply" becomes about 7 hours long. (40t/s vs 1t/s).

I don't really do anything at my job that could be done in a single prompt, and certainly not something that would take me 7 hours. A 7 hour task would take multiple iterations with how LLMs are right now.

You said it would do overnight what would take a human half a week, so I'm curious what tasks you are doing where AI is 3x faster than you even at 1t/s

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#67

Post by copperx »

Izmaki wrote: post #55
redox99 wrote: post #42

Earlier quoted context omitted.

At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.

This is what people forget when they see slow performance: at 1 t/s it's still roughly the equivalent of having another person work for you at no extra cost besides the initial purchase/sign-on-bonus. Frontier models are amazing, but what will really be useful for us is having models and hardware so efficient that you can run useful LLMs locally. One of my favourite LLMs to this day is still my jail-broken gemma4 12b…

How are you running it, if I may ask?

Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs

#70

Post by redox99 »

Capricorn2481 wrote: post #66
redox99 wrote: post #63

Earlier quoted context omitted.

Anything you do right now? A typical 10 minute prompt "simply" becomes about 7 hours long. (40t/s vs 1t/s).

I don't really do anything at my job that could be done in a single prompt, and certainly not something that would take me 7 hours. A 7 hour task would take multiple iterations with how LLMs are right now. You said it would do overnight what would take a human half a week, so I'm curious what tasks you are doing where AI is 3x faster than you even at 1t/s

Your prompts are probably very underspecified then. Frontier models one shot the majority of my prompts. UI is kind of the exception, there I do have to ask for a lot of tweaks.
Post reply on HN