Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MCP in LM Studio

lmstudio.ai

Post reply on HN
Print view View original

141–150 of 156 posts

Re: MCP in LM Studio

#141

Post by prophesi »

diggan wrote: post #130
prophesi wrote: post #120

Earlier quoted context omitted.

I believe they're talking about the rumors by an Apple supply chain analyst, Ming-Chi Kuo. https://www.techspot.com/news/106159-apple-m5-silicon-rumore...

Seems Apple is waking up to the fact that if it's too easy to run weights locally, there really isn't much sense to having their own remote inference endpoints, so time to stop the party :)

I thought their goal was to completely remove the need for a remote inference endpoint in the first place? May have read your comment wrong.

Re: MCP in LM Studio

#142

Post by pxc »

vntok wrote: post #136
diggan wrote: post #109

Earlier quoted context omitted.

> m4 with 24gb Wow, that is probably analogous to 48GB on other systems then, if we were to ask an Apple VP?

Not sure what Apple VPs have to do with the tech but yeah, pretty much any core engineer you ask at Apple will tell you this. Here is a nice article with some info about what memory compression is and how it works: https://arstechnica.com/gadgets/2013/10/os-x-10-9/#page-17 It's been a hard technical problem but is pretty much solved by now since its first debut in 2012-2013.

I've heard good things about how macOS handles memory relative to other operating systems. But Linux and Windows both have memory compression nowadays. So the claim is then not that memory compression makes your RAM twice as effective, but that macOS' memory compression is twice as good as the real and existing memory compression available on other operating systems.

Doesn't such a claim... need stronger evidence?

Re: MCP in LM Studio

#143

Post by diggan »

prophesi wrote: post #141
diggan wrote: post #130

Earlier quoted context omitted.

Seems Apple is waking up to the fact that if it's too easy to run weights locally, there really isn't much sense to having their own remote inference endpoints, so time to stop the party :)

I thought their goal was to completely remove the need for a remote inference endpoint in the first place? May have read your comment wrong.

No, I think Apple been clear from the beginning that they won't be able to do everything on the devices themselves, that's why they're building the infrastructure/software for their "cloud intelligence system" or whatever they call it.

Re: MCP in LM Studio

#144

Post by diggan »

remram wrote: post #135
kreetx wrote: post #133

Earlier quoted context omitted.

Using LLMs is entirely fine, but poor review for a protocol definition is ..degenerate. Aren't protocols supposed to be precise?

It was written by one vendor for their own use. It is miles away from an RFC or "standard"

Regardless if it's a RFC, standard or whatever, protocols need to be precise, exact and correct. And I think they wrote MCP with the idea of others using it, otherwise why even make it public if it's just for their own usage?

Re: MCP in LM Studio

#145

Post by storus »

diggan wrote: post #108
storus wrote: post #104

Earlier quoted context omitted.

RTX Pro 6000 can't do DeepSeek R1 671B Q4, you'd need 5-6 of them, which makes it way more expensive. Moreover, MacStudio will do it at 150W whereas Pro 6000 would start at 1500W.

> Moreover, MacStudio will do it at 150W whereas Pro 6000 would start at 1500W. No, Pro 6000 pulls max 600W, not sure where you get 1500W from, that's more than double the specification. Besides, what is the token/second or second/token, and prompt processing speed for running DeepSeek R1 671B on a Mac Studio with Q4? Curious about those numbers, because I have a feeling they're very far off each other.

You need at least 5x Pro 6000 (for smaller contexts), let's say Max-Q edition running at 300W, so overall you get a minimum of 1500W.

You get around 6 tokens/second which is not great but not terrible. If you use very long prompts, things get bad.

Re: MCP in LM Studio

#146

Post by xyc »

ijk wrote: post #94
xyc wrote: post #52

Earlier quoted context omitted.

It's a protocol that doesn't dictate how you are calling the tool. You can use in-memory transport without needing to spin up a server. Your tool can just be a function, but with the flexibility of serving to other clients.

Are there any examples of that? All the documentation I saw seemed to be about building an MCP server, with very little about connecting an existing inference infrastructure to local functions.

For TypeScript you can refer to https://github.com/modelcontextprotocol/typescript-sdk/blob/...

There isn't much documentation available right now but you can ask coding agent eg. Claude Code to generate an example.

Re: MCP in LM Studio

#147

Post by SparkyMcUnicorn »

diggan wrote: post #107
t1amat wrote: post #62

Earlier quoted context omitted.

The UI is the product. If you just want the engine, use mlx-omni-server (for MLX) or llama-swap (for GGUF) and huggingface-cli (for model downloads).

Those don't offer the same features as LM Studio itself does, even when you don't consider the UI. If there was a "LM Engine" CLI I could install, then yeah, but there isn't, hence the need to run the UI once to get "the engine".

I haven't used this and maybe it doesn't solve the problem you're describing, but might be worth looking at.

https://github.com/lmstudio-ai/lms

https://lmstudio.ai/docs/cli

Re: MCP in LM Studio

#148

Post by 112233 »

pests wrote: post #97
zackify wrote: post #70

Earlier quoted context omitted.

Because waiting forever for initial prompt processing with realistic number of MCP tools enabled on a prompt is going to suck without the most bandwidth possible And you are never going to sit around waiting for anything larger than the 96+gb of ram that the RTX pro has. If you’re using it for background tasks and not coding it’s a different story

Initial prompt processing with a large static context (system prompt + tools + whatever) could technically be improved by checkpointing the model state and reusing for future prompts. Not sure if any tools support this.

Dropping in late into this discussion, but is there any way to "comfortably" use multiple precomputed kv-caches with current models, in the style of this work: https://arxiv.org/abs/2212.10947 ?

Meaning, I pre-parse multiple documents, and the prompt and completion attention sees all of them, but there is no attention between the documents (they are all encoded in the same overlapping positions).

This way you can include basically unlimited amount of data in the prompt, paying for it with the perfomance.

Re: MCP in LM Studio

#149

Post by 112233 »

marci wrote: post #55
zackify wrote: post #36

Earlier quoted context omitted.

I love LM studio but I’d never waste 12k like that. The memory bandwidth is too low trust me. Get the RTX Pro 6000 for 8.5k with double the bandwidth. It will be way better

You can't run deepseek-v3/r1 on the RTX Pro 6000, not to mention the upcomming 1 million context qwen models, or the current qwen3-235b.

I can run full deepseek r1 on m1 max with 64GB of ram. Around 0.5 t/s with small quant. Q4 quant of Maverick (253 GB) runs at 2.3 t/s on it (no GPU offload).

Practically, last gen or even ES/QS EPYC or Xeon (with AMX), enough RAM to fill all 8 or 12 channels plus fast storage (4 Gen5 NVMEs are almost 60 GB/s) on paper at least look like cheapest way to run these huge MoE models at hobbyist speeds.

Re: MCP in LM Studio

#150

Post by Eupolemos »

boredemployee wrote: post #68
nix0n wrote: post #8

Earlier quoted context omitted.

LM Studio is quite good on Windows with Nvidia RTX also.

care to elaborate? i have rtx 4070 12gb vram + 64gb ram, i wonder what models I can run with it. Anything useful?

If you go to huggingface.co, you can tell it what specs you have and when you go to a model, it'll show you what variations of that model are likely to run well.

So if you go to this[0] random model, on the right there is a list of quantifications based on bits, and those you can run will be shown in green.

[0] https://huggingface.co/unsloth/Mistral-Small-3.1-24B-Instruc...

Post reply on HN