60s/token - if only there was a way to drop that "s" this would be amazing
Running Kimi K3 on a M1 Max
41–50 of 97 posts
Re: Running Kimi K3 on a M1 Max
#42[dead]
Re: Running Kimi K3 on a M1 Max
#43Now set it up with an agent and a permanent `/goal` to say it cannot stop until it has solved for speed, then leave it on and livestream so we can all see when it becomes exponential. Could have the Eternal Jukebox playing in the background!
That sounds like the most boring exciting livestream of all time.
Re: Running Kimi K3 on a M1 Max
#44Why not train another smaller LLM to give the same answers as Kimi K3?
Re: Running Kimi K3 on a M1 Max
#45Says it requires a 2TB disk? Must it be internal NVMe?
You can use an external drive if it's mounted as a writable volume. I would make sure it's fast, maybe thunderbolt 3/4/5 enclosure with a fast drive.
FYI, Thunderbolt 3 NVMe enclosures will be 10Gbps and 4 will be 40Gbps which is a big difference (and you'll notice it in the pricing as well)
Re: Running Kimi K3 on a M1 Max
#46SSD streaming on an M5 Max 128GB: https://x.com/antirez/status/2082136334160818528 Soon decent speed across two Mac Studios with 512GB of RAM.
0.3tx per second is decent speed?
And it gets worse with every token.
Re: Running Kimi K3 on a M1 Max
#47Earlier quoted context omitted.
So you subscribe to the belief we won't in future find mentalism in other galaxies or solar systems which operate on mechanisms we don't understand and think v e r y s l o w w w w w w w l y ? (note. I am not a believer in AGI) "useful" is highly contextual. The clock of the long "now" is not useful in the sense you mean, to synchronise your wristwatch. I'm still glad it exists.
Are there any well thought through stories about what this would look like? For example, I'm thinking about like nutrient flow, decision making, energy input, gravitational force, things like that seem to govern the value and speed of intelligence.
For the opposite, "Dragon's Egg" by Robert L. Forward is a fantastic read
Re: Running Kimi K3 on a M1 Max
#480.01 tk/s is unusable for anything, you would wait a whole day for just 1000 token of output, what is the point of projects like this?
I can only guess some post purchase remorse.
Need to justify buying an expensive rig that doesn't do what you expected.
Specifically thinking the people they could do something AI with cpu, and realizing it isn't feasible. Happened at my fortune 20 company. They had to get approvals and ofc it was useless. Plenty people tried to explain, but they were the principle engineer, and out ranked everyone.
"It's not going to work", the topic changed, and we never spoke about it again.
Re: Running Kimi K3 on a M1 Max
#490.01 tk/s is unusable for anything, you would wait a whole day for just 1000 token of output, what is the point of projects like this?
It is fun. Also its answering the question of what gonna happen if you wake up tomorrow and datacenters are gone. Or internets are gone. Some people on our globe live in countries with no internet whatsoever. Of course most of them dont have Macbook with 64GB RAM either, but it's much much easier to get than internet connection or rack of GB200. SOTA LLMs are efficiently compression of all the knowkedge humanity has…
Even if somehow all the data centers are gone, it's still uselessly slow.
But also that's a pretty extreme hypothetical. Imagine the polymarket on that.