MCP in LM Studio • Hacker News Board

Re: MCP in LM Studio

#151

Post by chisleu » Sat, Jun 28, 2025, 7:47 PM UTC

whatevsmate wrote: post #119

chisleu wrote: post #2

Just ordered a $12k mac studio w/ 512GB of integrated RAM. Can't wait for it to arrive and crank up LM Studio. It's literally the first install. I'm going to download it with safari. LM Studio is newish, and it's not a perfect interface yet, but it's fantastic at what it does which is bring local LLMs to the masses w/o them having to know much. There is another project that people should be aware of: https://github.c…

I did this a month ago and don't regret it one bit. I had a long laundry list of ML "stuff" I wanted to play with or questions to answer. There's no world in which I'm paying by the request, or token, or whatever, for hacking on fun projects. Keeping an eye on the meter is the opposite of having fun and I have absolutely nowhere I can put a loud, hot GPU (that probably has "gamer" lighting no less) in my fam's small…

Right on. I also have a laundry list of ML things I want to do starting with fine tuning models.

I don't mind paying for models to do things like code. I like to move really fast when I'm coding. But for other things, I just didn't want to spend a week or two coming up on the hardware needed to build a GPU system. You can just order a big GPU box, but it's going to cost you astronomically right now. Building a system with 4-5 PCIE 5.0 x16 slots, enough power, enough pcie lanes... It's a lot to learn. You can't go on PC part picker and just hunt a motherboard with 6 double slots.

This is a machine to let me do some things with local models. My first goal is to run some quantized version of the new V3 model and try to use it for coding tasks.

I expect it will be slow for sure, but I just want to know what it's capable of.

Re: MCP in LM Studio

#152

Post by chisleu » Sat, Jun 28, 2025, 7:48 PM UTC

datpuz wrote: post #138

chisleu wrote: post #2

Just ordered a $12k mac studio w/ 512GB of integrated RAM. Can't wait for it to arrive and crank up LM Studio. It's literally the first install. I'm going to download it with safari. LM Studio is newish, and it's not a perfect interface yet, but it's fantastic at what it does which is bring local LLMs to the masses w/o them having to know much. There is another project that people should be aware of: https://github.c…

I genuinely cannot wrap my head around spending this much money on hardware that is dramatically inferior to hardware that costs half the price. MacOS is not even great anymore, they stopped improving their UX like a decade ago.

How can you say something so brave, and so wrong?

Re: MCP in LM Studio

#153

Post by chisleu » Sat, Jun 28, 2025, 7:51 PM UTC

zackify wrote: post #36

chisleu wrote: post #2

Just ordered a $12k mac studio w/ 512GB of integrated RAM. Can't wait for it to arrive and crank up LM Studio. It's literally the first install. I'm going to download it with safari. LM Studio is newish, and it's not a perfect interface yet, but it's fantastic at what it does which is bring local LLMs to the masses w/o them having to know much. There is another project that people should be aware of: https://github.c…

I love LM studio but I’d never waste 12k like that. The memory bandwidth is too low trust me. Get the RTX Pro 6000 for 8.5k with double the bandwidth. It will be way better

Only on HN can buying a $12k badass computer be a waste of money

Re: MCP in LM Studio

#154

Post by chisleu » Sat, Jun 28, 2025, 7:55 PM UTC

zackify wrote: post #70

tymscar wrote: post #56

Earlier quoted context omitted.

Why would they pay 2/3 of the price for something with 1/5 of ram? The whole point of spending that much money for them is to run massive models, like the full R1, which the Pro 6000 cant

Because waiting forever for initial prompt processing with realistic number of MCP tools enabled on a prompt is going to suck without the most bandwidth possible And you are never going to sit around waiting for anything larger than the 96+gb of ram that the RTX pro has. If you’re using it for background tasks and not coding it’s a different story

You are correct that inference speed per $ is not optimized with this purchase.

What is optimized is the ability to find tune medium size models (~200GB) / $

You just can't get 500GB of VRAM for less than $100k. Even with $9k Blackwell cards, you have $10k in a barebones GPU server. You can't use commodity hardware and cluster it because you need fast interconnects. I'm talking 200-400GB/s interconnects. And those take yet another PCIe slot and require expensive Infiniband switches.

Shit gets costly fast. I consternated about this purchase for weeks. Eventually deciding that it's the easiest path to success for my purposes. Not for everyone's, but for mine.

Re: MCP in LM Studio

#155

Post by chisleu » Sat, Jun 28, 2025, 7:58 PM UTC

diggan wrote: post #126

smcleod wrote: post #117

Earlier quoted context omitted.

RTX is nice, but it's memory limited and requires to have a full desktop machine to run it in. I'd take slower inference (as long as it's not less than 15tk/s) for more memory any day!

I'd love to see more Very-Large-Memory Mac Studio benchmarks for prompt processing and inference. The few benchmarks I've seem either missed to take prompt processing into account, didn't share exact weights+setup that were used or showed really abysmal performance.

Oh I plan to produce a ton of that. I'll post a blog on it to HN and /r/localllama when I'm done.

Re: MCP in LM Studio

#156

Post by marci » Thu, Jul 03, 2025, 1:48 PM UTC

112233 wrote: post #149

marci wrote: post #55

Earlier quoted context omitted.

You can't run deepseek-v3/r1 on the RTX Pro 6000, not to mention the upcomming 1 million context qwen models, or the current qwen3-235b.

I can run full deepseek r1 on m1 max with 64GB of ram. Around 0.5 t/s with small quant. Q4 quant of Maverick (253 GB) runs at 2.3 t/s on it (no GPU offload). Practically, last gen or even ES/QS EPYC or Xeon (with AMX), enough RAM to fill all 8 or 12 channels plus fast storage (4 Gen5 NVMEs are almost 60 GB/s) on paper at least look like cheapest way to run these huge MoE models at hobbyist speeds.

If you're talking about Deepseek r1 with llama.cpp and mmap, then at this point you can run deepseek r1 on a raspberry zero with a 256GB micro sdcard and a phone charger. The only metric left to know is one's patience.