Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LM Studio – Discover, download, and run local LLMs

lmstudio.ai

Post reply on HN
Print view View original

41–50 of 157 posts

Re: LM Studio – Discover, download, and run local LLMs

#41

Post by pentagrama »

Curious about this and I just download it.

Want to try uncensored models.

I have a question, looking for the most popular "uncensored" model I just find "TheBloke/Luna-AI-Llama2-Uncensored-GGML", but it has 14 files to download between 2 to 7 GB, I just download the first one: https://imgur.com/a/DE2byOB

I try the model and it works: https://imgur.com/a/2vtPcui

I should download all the 14 files to get better results?

Also, asking how to make a bomb it looks that at least this model isn't "uncesored": https://imgur.com/a/iYz7VYQ

Re: LM Studio – Discover, download, and run local LLMs

#42

Post by smcleod »

hugovie wrote: post #33

LMStudio is great to run local LLMs, also support OpenAI-compatible API. In the case you need more advance UI/UX, you can use LMStudio with MindMac( https://mindmac.app ), just check this video for details https://www.youtube.com/watch?v=3KcVp5QQ1Ak .

Thanks for sharing MindMac - just tried it out and it's exactly what I was looking for, great to see Ollama integration is coming soon!

Re: LM Studio – Discover, download, and run local LLMs

#43

Post by jwblackwell »

pentagrama wrote: post #41

Curious about this and I just download it. Want to try uncensored models. I have a question, looking for the most popular "uncensored" model I just find "TheBloke/Luna-AI-Llama2-Uncensored-GGML", but it has 14 files to download between 2 to 7 GB, I just download the first one: https://imgur.com/a/DE2byOB I try the model and it works: https://imgur.com/a/2vtPcui I should download all the 14 files to get better results…

basically no open source fine tunes are censored, you can get an idea of popular models people are using here: https://openrouter.ai/models?o=top-weekly

teknium/openhermes-2.5-mistral-7b is a good one

You don't need all 14 files, just pick one that is recommended with a slight loss of quality - hover over the little (i)icon to find out

Re: LM Studio – Discover, download, and run local LLMs

#45

Post by eurekin »

easygenes wrote: post #14
eurekin wrote: post #7

This app could use some simple UI improvements: - The chatbox field has a normal "write here" state, when no chat is really selected. I thought my keyboard broke until I discovered that - I didn't find a way to set cuda acceleration before loading a model, only managed to set gpu offloaded layers and using "relaunch to apply" - Some HugginFace models are simply not listed and there's no indication about why. I guess…

It’s basically a front end for llama.cpp, so it will only show models with GGUF quantizations.

Ah, makes sense

Re: LM Studio – Discover, download, and run local LLMs

#46

Post by eurekin »

hahn-kev wrote: post #15
eurekin wrote: post #7

This app could use some simple UI improvements: - The chatbox field has a normal "write here" state, when no chat is really selected. I thought my keyboard broke until I discovered that - I didn't find a way to set cuda acceleration before loading a model, only managed to set gpu offloaded layers and using "relaunch to apply" - Some HugginFace models are simply not listed and there's no indication about why. I guess…

Those are really weird bugs, how do you even manage that these days

Not sure if sarcastic or not. Assuming not.

I did find it quite useful for opening a socket for remote tooling (played around withe the continue plugin).

The quirky UI did slow me down,but nothing really showstopping

Re: LM Studio – Discover, download, and run local LLMs

#47

Post by transcriptase »

3cats-in-a-coat wrote: post #25
andy99 wrote: post #17

I don't mean this as a criticism, I'm just curious because I work in this space too: who is this for? What is the niche of people savvy enough to use this who can't run one of the many open source local llm software? It looks in the screenshot like it's exposing much of the complexity of configuration anyway. Is the value in the interface and management of conversation and models? It would be nice to see info or even…

It's for people who want to discover LLMs and either don't have the skill to deploy it, or value their time, and prefer not to fool around for hours getting it to work before they can try it. The fact it has configuration is good, as long as it has some defaults.

Exactly. People like me have been waiting for a tool like this.

I'm more than capable of compiling/installing/running pretty much any software, but all I want is the ability to chat with a LLM of my choice without spending an afternoon tabbing back to a 30 step esoteric GitHub .md full of caveats, assumptions, and requiring dependencies to be installed and configured according to preferences I don't have.

Re: LM Studio – Discover, download, and run local LLMs

#48

Post by nerdenough »

pentagrama wrote: post #41

Curious about this and I just download it. Want to try uncensored models. I have a question, looking for the most popular "uncensored" model I just find "TheBloke/Luna-AI-Llama2-Uncensored-GGML", but it has 14 files to download between 2 to 7 GB, I just download the first one: https://imgur.com/a/DE2byOB I try the model and it works: https://imgur.com/a/2vtPcui I should download all the 14 files to get better results…

The readme of their repositories each have tables that detail the quality of each file. The QK_4_M and QK_5_M seem to be the two main recommended ones for low quality loss while too being too large.

Only need 1 of the files, but recommend checking out the GGUF version of the model (just replace GGML in the URL) instead of GGML. Llama.cpp no longer supports GGML, and not sure if TheBloke still uploads new GGML versions of models.

Re: LM Studio – Discover, download, and run local LLMs

#49

Post by pentagrama »

jwblackwell wrote: post #43
pentagrama wrote: post #41

Curious about this and I just download it. Want to try uncensored models. I have a question, looking for the most popular "uncensored" model I just find "TheBloke/Luna-AI-Llama2-Uncensored-GGML", but it has 14 files to download between 2 to 7 GB, I just download the first one: https://imgur.com/a/DE2byOB I try the model and it works: https://imgur.com/a/2vtPcui I should download all the 14 files to get better results…

basically no open source fine tunes are censored, you can get an idea of popular models people are using here: https://openrouter.ai/models?o=top-weekly teknium/openhermes-2.5-mistral-7b is a good one You don't need all 14 files, just pick one that is recommended with a slight loss of quality - hover over the little (i)icon to find out

Thank you! Now I checked the (i) tooltips. Just downloaded the bigger file (7GB) that says "Minimal loss of quality".

Re: LM Studio – Discover, download, and run local LLMs

#50

Post by mfalcon »

rsolva wrote: post #28
bambax wrote: post #26

Earlier quoted context omitted.

I'd like to build myself a headless server to run models, that could be queried from various clients locally on my LAN, but am usure where to start and what the hardware requirements would be. Software can always be changed later but I'd rather buy the hardware parts only once. Do you have recommendations about this? or blog posts to get started? What would be a decent hardware configuration?

Ollama does this. I run it in a container on my homelab (Proxmox on a HP EliteDesk SFF G2 800) and 7B models run decently fast on CPU-only. Ollama has a nice API and makes it easy to manage models. Together with ollama-webui, it can replace ChatGPT 3.5 for most tasks. I also use it in VSCode and nvim with plugins, works great! I have been meaning to write a short blog post about my setup...

I've been trying Ollama locally. I've yet to know how it'll behave in a production setting.
Post reply on HN