Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LM Studio 0.4

lmstudio.ai

Post reply on HN
Print view View original

41–50 of 134 posts

Re: LM Studio 0.4

#41

Post by huydotnet »

anonym29 wrote: post #23
huydotnet wrote: post #15

I was hoping for the /v1/messages endpoint to use with Claude Code without any extra proxies :(

This is a breeze to do with llama.cpp, which has had Anthropic responses API support for over a month now. On your inference machine: you@yourbox:~/Downloads/llama.cpp/bin$ ./llama-server -m --alias --jinja --ctx-size 32768 --host 0.0.0.0 --port 8080 -fa on Obviously, feel free to change your port, context size, flash attention, other params, etc. Then, on the system you're running Claude Code on: export ANTHROPIC_BA…

yup, I've been using llama.cpp for that on my PC, but on my Mac I found some cases where MLX models work best. haven't tried MLX with llama.cpp, so not sure how that will work out (or if it's even supported yet).

Re: LM Studio 0.4

#42

Post by nubg »

jiqiren wrote: post #2

This release introduces parallel requests with continuous batching for high throughput serving, all-new non-GUI deployment option, new stateful REST API, and a refreshed user interface.

are parallel requests "free"? or do you half performance when sending two requests in parallel?

Re: LM Studio 0.4

#43

Post by metalliqaz »

biddit wrote: post #14
saberience wrote: post #6

What’s the main use-case for this? I get that I can run local models, but all the paid for (remote) models are superior. So is the use-case just for people who don’t want to use big tech’s models? Is this just for privacy conscious people? Or is this just for “adult” chats, ie porn bots? Not being cynical here, just wanting to understand the genuine reasons people are using it.

Yes, frontier models from the labs are a step ahead and likely will always be, but we've already crossed levels of "good enough for X" with local models. This is analogous to the fact that my iPhone 17 is technically superior to my iPhone 8, but my outcomes for text messaging are no better. I've invested heavily in local inference. For me, it's a mixture privacy, control, stability, cognitive security. Privacy - my a…

I just "invested heavily" (relatively modest, but heavy for me) in a PC for local inference. The RAM was painful. Anyway, for my focused programming tasks the 30B models are plenty good enough.

Re: LM Studio 0.4

#45

Post by PlatoIsADisease »

syntaxing wrote: post #4

I’m really excited for lmster and to try it out. It’s essentially what I want from ollama. Ollama has deviated so much from their original core principles. Ollama has been broken and slow to update model support. There’s this “vendor sync” I’ve been waiting (essentially update ggml) for weeks.

What was the original core principle of ollama?

I had used oobabooga back in the day and found ollama unnecessary.

Re: LM Studio 0.4

#47

Post by nunodonato »

mk89 wrote: post #20
tiderpenger wrote: post #9

Earlier quoted context omitted.

To justify investing a trillion dollars like everything else LLM-related. The local models are pretty good. Like I ran a test on R1 (the smallest version) vs Perplexity Pro and shockingly got better answers running on base spec Mac Mini M4. It's simply not true that there is a huge difference. Mostly it's hardcoded overoptimalization. In general these models aren't really becoming better.

I agree with this comment here. For me the main BIG deal is that cloud models have online search embedded etc, while this one doesn't. However, if you don't need that (e.g., translate, summarize text, writing code) probably is good enough.

you can do web searches in lm studio. just connect an mcp that does it. Serpapi has an mcp, for example

Re: LM Studio 0.4

#48

Post by PlatoIsADisease »

saberience wrote: post #6

What’s the main use-case for this? I get that I can run local models, but all the paid for (remote) models are superior. So is the use-case just for people who don’t want to use big tech’s models? Is this just for privacy conscious people? Or is this just for “adult” chats, ie porn bots? Not being cynical here, just wanting to understand the genuine reasons people are using it.

I originally used local models as a somewhat therapeutic/advice thing. I didn't want to give openAI all my dirt.

But then I decided I'm just a chemical reaction and a product of my environment, so I gave chatGPT all my dirt anyway.

But before, I cared about my privacy.

Re: LM Studio 0.4

#49

Post by dragonwriter »

saberience wrote: post #6

What’s the main use-case for this? I get that I can run local models, but all the paid for (remote) models are superior. So is the use-case just for people who don’t want to use big tech’s models? Is this just for privacy conscious people? Or is this just for “adult” chats, ie porn bots? Not being cynical here, just wanting to understand the genuine reasons people are using it.

> What’s the main use-case for this?

Running weights available models.

> I get that I can run local models, but all the paid for (remote) models are superior.

If that's clearly true for your use cases, then maybe this isn’t for you.

> So is the use-case just for people who don’t want to use big tech’s models?

Most weights available models are also “big tech’s”, or finetunes of them.

> Is this just for privacy conscious people? Or is this just for “adult” chats, ie porn bots?

Sure, those are among the use cases. And there can be very good reasons to be concerned about privacy in some applications. But they aren’t the only reasons.

There’s a diversity of weights-available models available, with a variety of specialized strengths. Sure, for general use, the big commercial models may generally be more capable, but they may not be optimal for all uses (especially when cost effectiveness is considered, given that capable weights-available models for some uses are very lightweight.)

Re: LM Studio 0.4

#50

Post by dragonwriter »

mk89 wrote: post #20
tiderpenger wrote: post #9

Earlier quoted context omitted.

To justify investing a trillion dollars like everything else LLM-related. The local models are pretty good. Like I ran a test on R1 (the smallest version) vs Perplexity Pro and shockingly got better answers running on base spec Mac Mini M4. It's simply not true that there is a huge difference. Mostly it's hardcoded overoptimalization. In general these models aren't really becoming better.

I agree with this comment here. For me the main BIG deal is that cloud models have online search embedded etc, while this one doesn't. However, if you don't need that (e.g., translate, summarize text, writing code) probably is good enough.

> For me the main BIG deal is that cloud models have online search embedded etc, while this one doesn't.

Models do not have online search embedded, they have tool use capabilities (possibly with specialized training for a web search tool), but that's true of many open and weights-available models, and they are run with harnesses that support tools and provide a web search tool (lmstudio is such a harness, and can easily be supplied with a web search tool.)

Post reply on HN