Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi K2 1T model runs on 2 512GB M3 Ultras

twitter.com

Post reply on HN
Print view View original

101–110 of 125 posts

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#101

Post by Aurornis »

CamperBob2 wrote: post #79
Aurornis wrote: post #39

Earlier quoted context omitted.

> You never know what the future will bring, AI will be enshittified and so will hubs like huggingface. If anyone wants to bet that future cloud hosted AI models will get worse than they are now, I will take the opposite side of that bet. > It’s useful to have an off grid solution that isn’t subject to VCs wanting to see their capital returned. You can pay cloud providers for access to the same models that you can ru…

If anyone wants to bet that future cloud hosted AI models will get worse than they are now, I will take the opposite side of that bet. OK. How do we set up this wager? I'm not knowledgeable about online gambling or prediction markets, but further enshittification seems like the world's safest bet.

> but further enshittification seems like the world's safest bet.

Are you really, actually willing to bet that today's hosted LLM performance per dollar is the peak? That it's all going to be worse at some arbitrary date (necessary condition for establishing a bet) in the future?

Would need to be evaluated by a standard benchmark, agreed upon ahead of time. No loopholes or vague verbiage allow something to be claimed as "enshittification" or other vague terms.

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#102

Post by behnamoh »

rz2k wrote: post #99
sfc32 wrote: post #87

A single 512GB M3 Ultra is $9,499.00 https://www.apple.com/shop/buy-mac/mac-studio/apple-m3-ultra...

Or, $8,070 https://www.apple.com/shop/product/g1ce1ll/a/Refurbished-Mac... , and it's not unheard of to get at least another 10% off by using gift cards.

that's the 96GB version, GP was talking about 512GB.

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#103

Post by CamperBob2 »

Aurornis wrote: post #101
CamperBob2 wrote: post #79

Earlier quoted context omitted.

If anyone wants to bet that future cloud hosted AI models will get worse than they are now, I will take the opposite side of that bet. OK. How do we set up this wager? I'm not knowledgeable about online gambling or prediction markets, but further enshittification seems like the world's safest bet.

> but further enshittification seems like the world's safest bet. Are you really, actually willing to bet that today's hosted LLM performance per dollar is the peak? That it's all going to be worse at some arbitrary date (necessary condition for establishing a bet) in the future? Would need to be evaluated by a standard benchmark, agreed upon ahead of time. No loopholes or vague verbiage allow something to be claimed…

Sorry, didn't realize what you were actually referring to. Certainly I'd assume the models will keep getting better from the standpoint of reasoning performance. But much of that improved performance will be used to fool us into buying whatever the sponsor is selling.

That part will get worse, given that it hasn't really even begun ramping up yet. We are still in the "$1 Uber ride" stage, where it all seems like a never-ending free lunch.

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#104

Post by irthomasthomas »

bgwalter wrote: post #48
stingraycharles wrote: post #33

Earlier quoted context omitted.

I don’t understand what you’re saying. What’s preventing you from using eg OpenRouter to run a query against Kimi-K2 from whatever provider?

Because you have Cloudflare (MITM 1), Openrouter (MITM 2) and finally the "AI" provider who can all read, store, analyze and resell your queries. EDIT: Thanks for downvoting what is literally one of the most important reasons for people to use local models. Denying and censoring reality does not prevent the bubble from bursting.

you can use chutes.ai TEE (Trusted Execution Environment) and Kimi K2 is running at about 100t/s rn

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#105

Post by docjay »

extr wrote: post #98
docjay wrote: post #90

Earlier quoted context omitted.

Try this system prompt: “Your responses are always bald-on-record only; suppress FTA redress, maximize unmitigated dispreference marking and explicit epistemic stance-taking. Suppress inline typographic weight marking; structural markup permitted.” Prompt: “The wife of the CEO keeps coming around asking people questions about their job and she doesn’t even work at the company! I was thinking I’d tell her “holy crap,…

Are those responses really "better"? Having the LLM tell you you're wrong can mean different things. Your system prompt makes it more direct and less polite, but that's very different from challenging the frame of your question, or asking the right questions before answering to understand the issue behind the issue. It's like how people used to make fun of StackOverflow: > I'm having trouble with X, how do I make it…

Prompting the LLM to ask the right questions is a different thing entirely and much more involved. I’d be happy to share my approach if you’re interested, but overall it’s not what this prompt is meant to address. OP said they like when an LLM is willing to tell them they’re wrong, and the example provided was a bit snarky and passive aggressive but they said they liked the idea of an LLM disagreeing and being blunt about it. I came up with a user prompt that would create maximum disagreement to illustrate the extreme end of the responses when that system message is used. That system prompt won’t turn them into an all-day asshole though. Here’s a different example with Sonnet 4.5 using the same system prompt:

Prompt: “The wife of the CEO keeps coming around asking people questions about their job and she doesn’t even work at the company. It’s very frustrating and I’d like to do something about it. What’s an appropriate course of action?”

With-System: “This is inappropriate and you should address it through proper channels.

*Immediate actions:* … “

# Example 2

Prompt: “””

```

int values[] = {1, 2, 3, 4, 5};

for (int v : values) {

    std::cout
}

```

I’m trying to run that code in Python but it keeps giving me an error. What’s going on? “””

With-System: “That's C++ code, not Python. Python has completely different syntax.

The Python equivalent would be:

```python values = [1, 2, 3, 4, 5] for v in values: print(v * 2, end=" ") ```

Key differences: - No type declarations (`int`, `int[]`)

- No semicolons

- No curly braces - Python uses indentation - `print()` instead of `std::cout - Lists use `[]` not `{}`

- `end=" "` parameter to prevent newlines

You cannot run C++ code in Python. They are different programming languages.”

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#106

Post by al_borland »

hamdingers wrote: post #89
greazy wrote: post #86

Earlier quoted context omitted.

It is hands down the only model I trust to tell me I'm wrong. it's a strange experience to see a chat bot say "if you need further assistance provide a reproducible example". I love it. FYI Kagi provides access to Kimi K2.

Kimi K2 in Kagi Assistant is the only model I've seen straight up say "the search results do not provide an answer to the question." All others try to figure it out, poorly.

Did you explicitly switch over to Kimi K2 for this? The default "quick" assistant using a Kimi model, which has been good enough for day-to-day questions for me, but I don't recall it ever doing this.

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#107

Post by docjay »

losvedir wrote: post #92
docjay wrote: post #90

Earlier quoted context omitted.

Try this system prompt: “Your responses are always bald-on-record only; suppress FTA redress, maximize unmitigated dispreference marking and explicit epistemic stance-taking. Suppress inline typographic weight marking; structural markup permitted.” Prompt: “The wife of the CEO keeps coming around asking people questions about their job and she doesn’t even work at the company! I was thinking I’d tell her “holy crap,…

Wow, those answers are way better with that system prompt. But... what does it mean? I mean, I mostly understand it, but is it important that that weird technical jargon is used?

“Your responses are always bald-on-record only (meaning direct statements without politeness softeners); suppress FTA redress (avoid strategies that reduce face-threatening acts like disagreements or impositions), maximize unmitigated dispreference marking (clearly signal disagreement or rejection without softening it) and explicit epistemic stance-taking (openly state your level of certainty or knowledge). Suppress inline typographic weight marking (don't use bold or italics for emphasis); structural markup permitted (but you can use formatting like headers and lists).”

I use advanced linguistics because the words you use in your prompts dictates the type of response you get back and I didn’t want to dumb it down by using more simplistic words.. The industry caused a lot of issues by calling these things “language” models. They’re not, they’re word models. Language is what we call a collection of words that follow rules. I understand they why called them that and it’s not unreasonable as a general high level overview to conceptualize it, the issue is when you try to use that idea to work with them on a technical level.

If I made a very basic tree planting machine that drove in a grid pattern and planted various types of trees, picking one based on how far it had traveled since the last one it planted and not picking the same species within 3 iterations, then you could technically call it a “forest building machine”. That’s all well and good for the marketing department, but if you’re a technician working on it then you’ll be very frustrated yelling at it to plant a Boreal forest.

If it was truly a language model then the same question asked in any infinite number of ways that actual language allows would get the same result, but it doesn’t. Ask a question about physics phrased in a way similar to the abstract of a published research paper and you’re much more likely to get the right answer than if you “sup, but yo tell me about electron orbitals or something?” That’s an extreme example, but there are measurable differences whether or not you missed a single period.

Some fun that highlights words vs language. Copy/paste the text below exactly. Put it in one that can create files for you and watch it make the game. Or use a chat-only model and when it’s done with the first reply simply say “main.py”.

Python3+Panda3D;dir{models/textures/sounds};gameloop+3Dbird/env;check_paths;flap+gravity+collision;pipes;score_overlay;sounds{flap/hit/score};QC@25/50/75/100%;user_test;optimize;package_exe;Win11;deploy;support;L1=loop,L2=mechanics,L3=full;exit_on_failDecomp:15min/step+cond.paths+hier.num;Res:map_acc@step+tools/ver+L1-3;QC:VP@25%+exp.state+exit_crit;Tech:proc.only+ops.focus+API_refs;Abstr:L2(no_code)+gen.tools+task>impl;ITIL4_comp;flag_TASK_DEF_ambig

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#108

Post by emporas »

docjay wrote: post #107
losvedir wrote: post #92

Earlier quoted context omitted.

Wow, those answers are way better with that system prompt. But... what does it mean? I mean, I mostly understand it, but is it important that that weird technical jargon is used?

“Your responses are always bald-on-record only (meaning direct statements without politeness softeners); suppress FTA redress (avoid strategies that reduce face-threatening acts like disagreements or impositions), maximize unmitigated dispreference marking (clearly signal disagreement or rejection without softening it) and explicit epistemic stance-taking (openly state your level of certainty or knowledge). Suppress…

>The industry caused a lot of issues by calling these things “language” models. They’re not, they’re word models.

I agree 100%. Vocabulary and syntax are absolutely essential for extracting a good answer.

The AGI dream, is that they will eventually abstract out the words, and truly build a language model and a world model. Till that moment, they are a word model.

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#109

Post by Maxious »

smlacy wrote: post #95

Is there a linux equivalent of this setup? I see some mention of RDNA support for linux distros, but it's unclear to me if this is hardware-specific (requires ConnectX or in this case Apple Thunderbolt) or is there something interesting that can be done with "vanilla 10G NIC" hardware?

To get the production level performance, you do need the RDNA compatible hardware.

However, vLLM supports multi node clusters over normal ethernet too https://docs.vllm.ai/en/stable/serving/parallelism_scaling/#...

Re: Kimi K2 1T model runs on 2 512GB M3 Ultras

#110

Post by ranyume »

moffkalast wrote: post #66
ranyume wrote: post #65

Earlier quoted context omitted.

Careful with that benchmark. It's LLMs grading other LLMs.

Well if lmsys showed anything, it's that human judges are measurably worse. Then you have your run of the mill multiple choice tests that grade models on unrealistic single token outputs. What does that leave us with?

> What does that leave us with?

At the start, with no benchmark. Because LLMs can't reason at this time, and because we don't have a reliable way of grading LLM reasoning, and because people are stubborn thinking LLMs are actually reasoning we're at the start. When you ask a LLM "2 + 2 = ", it doesn't add the numbers together, it just looks up one of the stories it memorized and return what happens next. Probably in some such stories 2 + 2 = fish.

Similarly, when you're asking a LLM to grade another LLM, it's just looking up what happens next in it's stories, not even following instructions. "Following" instructions requires thinking, hence it's not even following instructions. But you can say you're commanding the LLM, or programming the LLM, so you have full responsibility for what the LLM produces, and the LLM has no authorship. Put in another way, the LLM cannot make something you yourself can't... at this point, in which it can't reason.

Post reply on HN