The SSDs are necessary because Apple's architecture doesn't allow RAM upgrades. Reminds me of someone who said "640KB ought to be enough for anybody".
Apple uses SoC's which means the RAM is simply not upgradable because it is a SYSTEM on a CHIP
Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
131–140 of 159 posts
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#132Earlier quoted context omitted.
Maybe. Do you have an example of a prompt you've done recently? I don't think it's valuable at all to try and craft a specific prompt to "one-shot" a task, I'm way faster just doing small asks and guiding it. At 1t/s, how would you even know your prompt was insufficient? It would take you 7 hours to see it.
Stuff along the lines of implement controller service and tests for the following endpoints: - list of many endpoints with the JSON they receive and return and description of what they need to achieve Stuff you could probably do in a single work day if you lock in and enter flow state, but in a typical job takes like half a week. And the vast majority of times the AI one shots it with no bugs, where I would have copy…
Like I said, in my experience I would rather do this with a bunch of tiny prompts than try to craft something that oneshots it. Chances are it will do several things I don't want, no matter how clear I think I'm being.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#133Earlier quoted context omitted.
Like how crypto used ASICS but then didn't because the scaling of consumer hardware made it obsolete?
x86 has a built in instruction for doing AES. That's just moving the ASIC into the CPU core, not eliminating it.
Is there any cryptocurrency that uses AES?
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#134You currently can't run a 2.8T locally; there's just no way. So, it's a good start.
But my local is a 8xH100, you insensitive clod!
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#135Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#136Should LLMs be designed to be modular, so that instead of needing access to the whole model, for a given prompt, only a small subset of the model would be used? If knolwedge was sufficiently modularized, most of it could be ignored.
Maybe a hyopthetical model of 5T of indexable weights could be used with only 50 GB of GPU ram, efficiently, because it stays resident in the GPU.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#137Should LLMs be designed to be modular, so that instead of needing access to the whole model, for a given prompt, only a small subset of the model would be used? If knolwedge was sufficiently modularized, most of it could be ignored. Maybe a hyopthetical model of 5T of indexable weights could be used with only 50 GB of GPU ram, efficiently, because it stays resident in the GPU.
Isn't that the definition of an MoE model?
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#138Should LLMs be designed to be modular, so that instead of needing access to the whole model, for a given prompt, only a small subset of the model would be used? If knolwedge was sufficiently modularized, most of it could be ignored. Maybe a hyopthetical model of 5T of indexable weights could be used with only 50 GB of GPU ram, efficiently, because it stays resident in the GPU.
Isn't that the definition of an MoE model?
No, not really, current MoE limit the computation, not memory requirements. Router experts are not "sticky" enough to achieve what robrenaud describes - they'd have to be chosen per prompt, or at least per chunk, not per token.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#139Earlier quoted context omitted.
I would agree, but I want to add that I have real issues with combination of opencode plus slow inference (4-5tok/s). I get weird interruptions. I can only guess its related to some kind of timeouts in the harness or something. Its not a problem of the model of course, but it seems impractical atm. I wonder if anyone else had this kind of thing happening.
For OSs harnesses this seems like a good thing to point a paid model at fixing if you don't want to dig in yourself?
I might do that myself in the end
checked the bug reports and PRs first though, there are a few related, nothing merged yet
I ran opencode task again last night since it's slow and this morning i got a "SSE read timed out"
logs show tokens still streaming in steadily (every 3-4s, 46,818 tokens in) right up to the moment opencode disconnected
opencode cut a response that was actively generating (not a stalled one)
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#140Earlier quoted context omitted.
Isn't that the definition of an MoE model?
No, not really, current MoE limit the computation, not memory requirements. Router experts are not "sticky" enough to achieve what robrenaud describes - they'd have to be chosen per prompt, or at least per chunk, not per token.
What is "sticky" in this context?