Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K2 1T model runs on 2 512GB M3 Ultras

twitter.com

Post reply on HN
Print view View original

21–30 of 125 posts

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#21

Post by IgorPartola »

stingraycharles wrote: post #12
Kim_Bruning wrote: post #9

Earlier quoted context omitted.

> Isn’t “instruction following” the most important thing you’d want out of a model in general, No. And for the same reason that pure "instruction following" in humans is considered a form of protest/sabotage. https://en.wikipedia.org/wiki/Work-to-rule

I don’t understand the point you’re trying to make. LLMs are not humans. From my perspective, the whole problem with LLMs (at least for writing code) is that it shouldn’t assume anything, follow the instructions faithfully, and ask the user for clarification if there is ambiguity in the request. I find it extremely annoying when the model pushes back / disagrees, instead of asking for clarification. For this reason,…

Full instruction following looks like monkey’s paw/malicious compliance. A good way to eliminate a bug from a codebase is to delete the codebase, that type of thing. You want the model to have enough creative freedom to solve the problem otherwise you are just coding using an imprecise language spec.

I know what you mean: a lot of my prompts include “never use em-dashes” but all models forget this sooner or later. But in other circumstances I do want it to push back on something I am asking. “I can implement what you are asking but I just want to confirm that you are ok with this feature introducing an SQL injection attack into this API endpoint”

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#22

Post by Kim_Bruning »

stingraycharles wrote: post #12
Kim_Bruning wrote: post #9

Earlier quoted context omitted.

> Isn’t “instruction following” the most important thing you’d want out of a model in general, No. And for the same reason that pure "instruction following" in humans is considered a form of protest/sabotage. https://en.wikipedia.org/wiki/Work-to-rule

I don’t understand the point you’re trying to make. LLMs are not humans. From my perspective, the whole problem with LLMs (at least for writing code) is that it shouldn’t assume anything, follow the instructions faithfully, and ask the user for clarification if there is ambiguity in the request. I find it extremely annoying when the model pushes back / disagrees, instead of asking for clarification. For this reason,…

I can't help you then. You can find a close analogue in the OSS/CIA Simple Sabotage Field Manual. [1]

For that reason, I don't trust Agents (human or ai, secret or overt :-P) who don't push back.

[1] https://www.cia.gov/static/5c875f3ec660e092cf893f60b4a288df/... esp. Section 5(11)(b)(14): "Apply all regulations to the last letter." - [as a form of sabotage]

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#23

Post by andy99 »

chrsw wrote: post #13
websiteapi wrote: post #7

I get tempted to buy a couple of these, but I just feel like the amortization doesn’t make sense yet. Surely in the next few years this will be orders of magnitude cheaper.

The only reason why you run local models is for privacy, never for cost. Or even latency.

Autonomy generally, not just privacy. You never know what the future will bring, AI will be enshittified and so will hubs like huggingface. It’s useful to have an off grid solution that isn’t subject to VCs wanting to see their capital returned.

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#24

Post by lordswork »

stingraycharles wrote: post #10
websiteapi wrote: post #7

I get tempted to buy a couple of these, but I just feel like the amortization doesn’t make sense yet. Surely in the next few years this will be orders of magnitude cheaper.

I don’t think it will ever make sense; you can buy so much cloud based usage for this type of price. From my perspective, the biggest problem is that I am just not going to be using it 24/7. Which means I’m not getting nearly as much value out of it as the cloud based vendors do from their hardware. Last but not least, if I want to run queries against open source models, I prefer to use a provider like Groq or Cerebr…

As long as you're willing to wait up to an hour for your GPU to get scheduled when you do want to use it.

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#25

Post by [deleted] »

stingraycharles wrote: post #10
websiteapi wrote: post #7

I get tempted to buy a couple of these, but I just feel like the amortization doesn’t make sense yet. Surely in the next few years this will be orders of magnitude cheaper.

I don’t think it will ever make sense; you can buy so much cloud based usage for this type of price. From my perspective, the biggest problem is that I am just not going to be using it 24/7. Which means I’m not getting nearly as much value out of it as the cloud based vendors do from their hardware. Last but not least, if I want to run queries against open source models, I prefer to use a provider like Groq or Cerebr…

[deleted]

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#26

Post by websiteapi »

stingraycharles wrote: post #10
websiteapi wrote: post #7

I get tempted to buy a couple of these, but I just feel like the amortization doesn’t make sense yet. Surely in the next few years this will be orders of magnitude cheaper.

I don’t think it will ever make sense; you can buy so much cloud based usage for this type of price. From my perspective, the biggest problem is that I am just not going to be using it 24/7. Which means I’m not getting nearly as much value out of it as the cloud based vendors do from their hardware. Last but not least, if I want to run queries against open source models, I prefer to use a provider like Groq or Cerebr…

my issue is once you have it in your workflow I'd be pretty latency sensitive. imagine those record-it-all apps working well. eventually you'd become pretty reliant on it. I don't want to necessarily be at the whims of the cloud

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#27

Post by natrys »

geerlingguy wrote: post #11
Alifatisk wrote: post #2

You should mention that it is 4bit quant. Still very impressive!

Kiki K2 was made to be optimized at 4-bit, though.

That's the Kimi K2 Thinking, this post seems to be talking about original Kimi K2 Instruct though, I don't think INT4 QAT (quantization aware training) version was released for this.

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#28

Post by amelius »

wasting_time wrote: post #5
A_D_E_P_T wrote: post #3

Kimi K2 is a really weird model, just in general. It's not nearly as smart as Opus 4.5 or 5.2-Pro or whatever, but it has a very distinct writing style and also a much more direct "interpersonal" style. As a writer of very-short-form stuff like emails, it's probably the best model available right now. As a chatbot, it's the only one that seems to really relish calling you out on mistakes or nonsense, and it doesn't h…

It's also the only model that consistently nails my favorite AI benchmark: https://clocks.brianmoore.com/

But how sure are we that it wasn't trained on that specifically?

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#30

Post by wat10000 »

stingraycharles wrote: post #12
Kim_Bruning wrote: post #9

Earlier quoted context omitted.

> Isn’t “instruction following” the most important thing you’d want out of a model in general, No. And for the same reason that pure "instruction following" in humans is considered a form of protest/sabotage. https://en.wikipedia.org/wiki/Work-to-rule

I don’t understand the point you’re trying to make. LLMs are not humans. From my perspective, the whole problem with LLMs (at least for writing code) is that it shouldn’t assume anything, follow the instructions faithfully, and ask the user for clarification if there is ambiguity in the request. I find it extremely annoying when the model pushes back / disagrees, instead of asking for clarification. For this reason,…

If I tell it to fetch the information using HTPP, I want it to ask if I meant HTTP, not go off and try to find a way to fetch the info using an old printing protocol from IBM.
Post reply on HN