You should mention that it is 4bit quant. Still very impressive!
Kiki K2 was made to be optimized at 4-bit, though.
Kimi K2 1T model runs on 2 512GB M3 Ultras
11–20 of 125 posts
Re: Kimi K2 1T model runs on 2 512GB M3 Ultras
#12Earlier quoted context omitted.
> As a chatbot, it's the only one that seems to really relish calling you out on mistakes or nonsense, and it doesn't hesitate to be blunt with you. My experience is that Sonnet 4.5 does this a lot as well, but this is more often than not due to a lack of full context, eg accusing the user of not doing X or Y when it just wasn’t told that was already done, and proceeding to apologize. How is Kimi K2 in this regard? I…
> Isn’t “instruction following” the most important thing you’d want out of a model in general, No. And for the same reason that pure "instruction following" in humans is considered a form of protest/sabotage. https://en.wikipedia.org/wiki/Work-to-rule
I don’t understand the point you’re trying to make. LLMs are not humans.
From my perspective, the whole problem with LLMs (at least for writing code) is that it shouldn’t assume anything, follow the instructions faithfully, and ask the user for clarification if there is ambiguity in the request.
I find it extremely annoying when the model pushes back / disagrees, instead of asking for clarification. For this reason, I’m not a big fan of Sonnet 4.5.
Re: Kimi K2 1T model runs on 2 512GB M3 Ultras
#13I get tempted to buy a couple of these, but I just feel like the amortization doesn’t make sense yet. Surely in the next few years this will be orders of magnitude cheaper.
The only reason why you run local models is for privacy, never for cost. Or even latency.
Re: Kimi K2 1T model runs on 2 512GB M3 Ultras
#14Kimi K2 is a very impressive model! It's particularly un-obsequious, which makes it useful for actually checking your reasoning on things. Some especially older ChatGPT models will tell you that everything you say is fantastic and great. Kimi -on the other hand- doesn't mind taking a detour to question your intelligence and likely your entire ancestry if you ask it to be brutal.
Upon request cg roasts. Good for reducing distractions.
Re: Kimi K2 1T model runs on 2 512GB M3 Ultras
#15I get tempted to buy a couple of these, but I just feel like the amortization doesn’t make sense yet. Surely in the next few years this will be orders of magnitude cheaper.
The only reason why you run local models is for privacy, never for cost. Or even latency.
indeed - my main use case is those kind of "record everything" sort of setups. I'm not even super privacy conscious per se but it just feels too weird to send literally everything I'm saying all of the time to the cloud.
luckily for now whisper doesn't require too much compute, bu the kind of interesting analysis I'd want would require at least a 1B parameter model, maybe 100B or 1T.
Re: Kimi K2 1T model runs on 2 512GB M3 Ultras
#16I get tempted to buy a couple of these, but I just feel like the amortization doesn’t make sense yet. Surely in the next few years this will be orders of magnitude cheaper.
Hopefully the next time it’s updated, it should ship with some variant of the M5.
Re: Kimi K2 1T model runs on 2 512GB M3 Ultras
#17I get tempted to buy a couple of these, but I just feel like the amortization doesn’t make sense yet. Surely in the next few years this will be orders of magnitude cheaper.
I don’t think it will ever make sense; you can buy so much cloud based usage for this type of price. From my perspective, the biggest problem is that I am just not going to be using it 24/7. Which means I’m not getting nearly as much value out of it as the cloud based vendors do from their hardware. Last but not least, if I want to run queries against open source models, I prefer to use a provider like Groq or Cerebr…
I think you’re missing the whole point, which is not using cloud compute.
Re: Kimi K2 1T model runs on 2 512GB M3 Ultras
#18Earlier quoted context omitted.
> Isn’t “instruction following” the most important thing you’d want out of a model in general, No. And for the same reason that pure "instruction following" in humans is considered a form of protest/sabotage. https://en.wikipedia.org/wiki/Work-to-rule
I don’t understand the point you’re trying to make. LLMs are not humans. From my perspective, the whole problem with LLMs (at least for writing code) is that it shouldn’t assume anything, follow the instructions faithfully, and ask the user for clarification if there is ambiguity in the request. I find it extremely annoying when the model pushes back / disagrees, instead of asking for clarification. For this reason,…
I think the opposite. I don't want to write down everything and I like when my agents take some initiative or come up with solutions I didn't think of.
Re: Kimi K2 1T model runs on 2 512GB M3 Ultras
#19Earlier quoted context omitted.
> Isn’t “instruction following” the most important thing you’d want out of a model in general, No. And for the same reason that pure "instruction following" in humans is considered a form of protest/sabotage. https://en.wikipedia.org/wiki/Work-to-rule
I don’t understand the point you’re trying to make. LLMs are not humans. From my perspective, the whole problem with LLMs (at least for writing code) is that it shouldn’t assume anything, follow the instructions faithfully, and ask the user for clarification if there is ambiguity in the request. I find it extremely annoying when the model pushes back / disagrees, instead of asking for clarification. For this reason,…
I would assume that if the model made no assumptions, it would be unable to complete most requests given in natural language.
Re: Kimi K2 1T model runs on 2 512GB M3 Ultras
#20Earlier quoted context omitted.
> Isn’t “instruction following” the most important thing you’d want out of a model in general, No. And for the same reason that pure "instruction following" in humans is considered a form of protest/sabotage. https://en.wikipedia.org/wiki/Work-to-rule
I don’t understand the point you’re trying to make. LLMs are not humans. From my perspective, the whole problem with LLMs (at least for writing code) is that it shouldn’t assume anything, follow the instructions faithfully, and ask the user for clarification if there is ambiguity in the request. I find it extremely annoying when the model pushes back / disagrees, instead of asking for clarification. For this reason,…
> is that it shouldn’t assume anything, follow the instructions faithfully, and ask the user for clarification if there is ambiguity in the request
We already had those. They are called programming languages. And interacting with them used to be a very well paid job.