LM Studio – Discover, download, and run local LLMs • Hacker News Board

Re: LM Studio – Discover, download, and run local LLMs

#51

Post by Loic » Wed, Nov 22, 2023, 2:22 PM UTC

rsolva wrote: post #28

bambax wrote: post #26

Earlier quoted context omitted.

I'd like to build myself a headless server to run models, that could be queried from various clients locally on my LAN, but am usure where to start and what the hardware requirements would be. Software can always be changed later but I'd rather buy the hardware parts only once. Do you have recommendations about this? or blog posts to get started? What would be a decent hardware configuration?

Ollama does this. I run it in a container on my homelab (Proxmox on a HP EliteDesk SFF G2 800) and 7B models run decently fast on CPU-only. Ollama has a nice API and makes it easy to manage models. Together with ollama-webui, it can replace ChatGPT 3.5 for most tasks. I also use it in VSCode and nvim with plugins, works great! I have been meaning to write a short blog post about my setup...

If I may ask, which plugins are you using in VSCode?

Re: LM Studio – Discover, download, and run local LLMs

#52

Post by hugovie » Wed, Nov 22, 2023, 2:22 PM UTC

smcleod wrote: post #42

hugovie wrote: post #33

LMStudio is great to run local LLMs, also support OpenAI-compatible API. In the case you need more advance UI/UX, you can use LMStudio with MindMac( https://mindmac.app ), just check this video for details https://www.youtube.com/watch?v=3KcVp5QQ1Ak .

Thanks for sharing MindMac - just tried it out and it's exactly what I was looking for, great to see Ollama integration is coming soon!

Thank you for your support. I just found a workaround solution to use Ollama with MindMac. Please check this video https://www.youtube.com/watch?v=bZfV70YMuH0 for more details. I will integrate Ollama deeply in the future version.

Re: LM Studio – Discover, download, and run local LLMs

#53

Post by addandsubtract » Wed, Nov 22, 2023, 2:32 PM UTC

omneity wrote: post #40

I really like LM Studio and had it open when I came across this post. LM Studio is an interesting mixture of: - A local model runtime - A model catalog - A UI to chat with the models easily - An openAI compatible API And it has several plugins such as for RAG (using ChromaDB) and others. Personally I think the positioning is very interesting. They're well positioned to take advantage of new capabilities in the OS eco…

Does it also let you connect to the ChatGPT API and use it?

Re: LM Studio – Discover, download, and run local LLMs

#54

Post by omneity » Wed, Nov 22, 2023, 2:38 PM UTC

addandsubtract wrote: post #53

omneity wrote: post #40

I really like LM Studio and had it open when I came across this post. LM Studio is an interesting mixture of: - A local model runtime - A model catalog - A UI to chat with the models easily - An openAI compatible API And it has several plugins such as for RAG (using ChromaDB) and others. Personally I think the positioning is very interesting. They're well positioned to take advantage of new capabilities in the OS eco…

Does it also let you connect to the ChatGPT API and use it?

I haven’t found that option. I know it exists in Gpt4all though.

Personally I use a locally served frontend to use ChatGPT via API.

Re: LM Studio – Discover, download, and run local LLMs

#55

Post by chatmasta » Wed, Nov 22, 2023, 2:47 PM UTC

smusamashah wrote: post #23

Why purple or some shade of purple is the color of all AI products? For some reason, the landing pages of AI products immediately remind of Crypto products. This one does not have Crypto vibes but the colour is purple. I don't get why.

It's a default color in Tailwind.css and is used in a lot of the templates and examples. Nine times out of ten, if you check the source of a page with this flavor of purple, you'll see it's using Tailwind, as the OP site in fact does.

Re: LM Studio – Discover, download, and run local LLMs

#56

Post by chatmasta » Wed, Nov 22, 2023, 3:13 PM UTC

Am I missing something here? I'm on a recent M2 machine. Every model I've downloaded fails to load immediately when trying to load it. Is there some way to get feedback on the reason for failure, like a log file or something?

EDIT: The problem is I'm on macOS 13.2 (Ventura). According to a message in Discord, the minimum version for some (most?) models is 13.6.

Re: LM Studio – Discover, download, and run local LLMs

#57

Post by thejohnconway » Wed, Nov 22, 2023, 3:16 PM UTC

manyoso wrote: post #36

For those looking for an open source alternative with Mac, Windows, Linux support check out GPT4All.io

Terrible name, given that its value is that it runs locally, and you can't do that with ChatGPT.

Re: LM Studio – Discover, download, and run local LLMs

#58

Post by ekianjo » Wed, Nov 22, 2023, 3:24 PM UTC

kgeist wrote: post #16

For my experiments with new self-hostable models on Linux, I've been using a script to download GGUF-models from TheBloke on HuggingFace (currently, TheBloke's repository has 657 models in the GGUF format) which I feed to a simple program I wrote which invokes llama.cpp compiled with GPU support. The GGUF format and TheBloke are a blessing, because I'm able to check out new models basically on the day of their releas…

You dont need thebloke. Its trivial to make gguf files from bin models by yourself.

Re: LM Studio – Discover, download, and run local LLMs

#59

Post by rgbrgb » Wed, Nov 22, 2023, 3:27 PM UTC

LMStudio is great, if a bit daunting. If you’re on Mac and want a native open source interface, try out FreeChat https://www.freechat.run

Re: LM Studio – Discover, download, and run local LLMs

#60

Post by cloudking » Wed, Nov 22, 2023, 3:29 PM UTC

Is anyone using open source models to actually get work done or solving problems in their software architecture? So far I haven't found anything near the quality of GPT-4.