Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running Kimi K3 on a M1 Max

github.com

Post reply on HN
Print view View original

51–60 of 97 posts

Re: Running Kimi K3 on a M1 Max

#51

Post by ggm »

tito wrote: post #32
ggm wrote: post #24

Earlier quoted context omitted.

So you subscribe to the belief we won't in future find mentalism in other galaxies or solar systems which operate on mechanisms we don't understand and think v e r y s l o w w w w w w w l y ? (note. I am not a believer in AGI) "useful" is highly contextual. The clock of the long "now" is not useful in the sense you mean, to synchronise your wristwatch. I'm still glad it exists.

Are there any well thought through stories about what this would look like? For example, I'm thinking about like nutrient flow, decision making, energy input, gravitational force, things like that seem to govern the value and speed of intelligence.

I think this is a slow version of the quandry behind Quantum Computing: how do you distinguish events from the noise floor? It happens in the quantum context and it would happen in the millenial timeframe completing "operations" which have to be compared to e.g. the stability of orbit around a sun.

Re: Running Kimi K3 on a M1 Max

#52

Post by brokencode »

tito wrote: post #35
brokencode wrote: post #28

Earlier quoted context omitted.

Maybe in terms of code produced, but one token is only a fragment of a thought for an LLM. It’d be like thinking as slowly as Ents talk to each other in Lord of the Rings.

Oh, is that how it works? So, when somebody says a model is running at X tokens per second, it means that the thinking process is running at that, and output tokens are much lower then? Thanks to the explanation.

It’s all just tokens to the model. Whether it’s thinking tokens or output tokens, they take the same amount of computation to produce. The only difference is whether the token is displayed to the user.

Re: Running Kimi K3 on a M1 Max

#54

Post by indiv0 »

tito wrote: post #32
ggm wrote: post #24

Earlier quoted context omitted.

So you subscribe to the belief we won't in future find mentalism in other galaxies or solar systems which operate on mechanisms we don't understand and think v e r y s l o w w w w w w w l y ? (note. I am not a believer in AGI) "useful" is highly contextual. The clock of the long "now" is not useful in the sense you mean, to synchronise your wristwatch. I'm still glad it exists.

Are there any well thought through stories about what this would look like? For example, I'm thinking about like nutrient flow, decision making, energy input, gravitational force, things like that seem to govern the value and speed of intelligence.

Hard to avoid spoilers here but Vernor Vinge hits on pretty much exactly this in A Fire Upon the Deep. Though he's interested less in the hard sci-fi aspects of how/why and more on the consequences of it (story-wise).

Re: Running Kimi K3 on a M1 Max

#55

Post by anigbrowl »

Azantys wrote: post #3

0.01 tk/s is unusable for anything, you would wait a whole day for just 1000 token of output, what is the point of projects like this?

'Large Language models? They can barely produce gibberish sentences, what would this tech ever be useful for?'

- bunch of people only ~4 years ago

Re: Running Kimi K3 on a M1 Max

#57

Post by mindwok »

Does anyone else feel like the writing is on the wall for a future of local models? Spamming data centres everywhere, powering them, having to commit insane capital to hardware, all the effort to serve inference over a network reliably - when here we are with a frontier model nearly running on a laptop.

Local AI on your device seems like a much more likely future to me than datacenters in space. For inference at least, training is another story.

Re: Running Kimi K3 on a M1 Max

#58

Post by api »

antirez wrote: post #4

SSD streaming on an M5 Max 128GB: https://x.com/antirez/status/2082136334160818528 Soon decent speed across two Mac Studios with 512GB of RAM.

I’ve wondered for a while: given the lower cost of SSD per GB could you build a very wide RAID0 style striped array of SSDs (maybe one per slot) to get almost RAM like read speeds?

To really go fast you’d probably have to do PCB layout and do like 256 or 1024 chips in parallel with a fast SRAM aggregation buffer feeding a GPU or TPU rig.

Or could you do the same with custom layout of cheap slower RAM?

I wonder if anyone is doing this? You would flash in a model and then just run it. It would need RAM for context but much less of it.

Re: Running Kimi K3 on a M1 Max

#60

Post by Azantys »

anigbrowl wrote: post #55
Azantys wrote: post #3

0.01 tk/s is unusable for anything, you would wait a whole day for just 1000 token of output, what is the point of projects like this?

'Large Language models? They can barely produce gibberish sentences, what would this tech ever be useful for?' - bunch of people only ~4 years ago

I questioned the speed not the output quality, that is another discussion
Post reply on HN