Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LM Studio – Discover, download, and run local LLMs

lmstudio.ai

Post reply on HN
Print view View original

61–70 of 157 posts

Re: LM Studio – Discover, download, and run local LLMs

#61

Post by voiceblue »

cloudking wrote: post #60

Is anyone using open source models to actually get work done or solving problems in their software architecture? So far I haven't found anything near the quality of GPT-4.

WizardCoder is probably close to state of the art for open models as of right now.

https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder

https://huggingface.co/WizardLM/WizardCoder-Python-34B-V1.0

Top of the line consumer machines can run this at a good clip, though most machines will need to use a quantized model (ExLlamaV2 is quite fast). I found a model for that as well, though I haven't used it myself:

https://huggingface.co/oobabooga/CodeBooga-34B-v0.1-EXL2-4.2...

Re: LM Studio – Discover, download, and run local LLMs

#62

Post by NoMoreNicksLeft »

rgbrgb wrote: post #59

LMStudio is great, if a bit daunting. If you’re on Mac and want a native open source interface, try out FreeChat https://www.freechat.run

Thanks for the link.

I expected it to not let me run this. I have an intel Macbook, was expecting that I'd need Apple Silicon... am I misunderstanding something? I get fairly fast results at the prompt with the default model. How's this thing running with whatever shitty GPU I have in my laptop?

Re: LM Studio – Discover, download, and run local LLMs

#63

Post by smusamashah »

chatmasta wrote: post #55
smusamashah wrote: post #23

Why purple or some shade of purple is the color of all AI products? For some reason, the landing pages of AI products immediately remind of Crypto products. This one does not have Crypto vibes but the colour is purple. I don't get why.

It's a default color in Tailwind.css and is used in a lot of the templates and examples. Nine times out of ten, if you check the source of a page with this flavor of purple, you'll see it's using Tailwind, as the OP site in fact does.

Ah! that makes more sense. New startup, new tech and therefore the new default color. I hope its just that and because I only tend to notice AI startups purple is what I end up seeing.

Re: LM Studio – Discover, download, and run local LLMs

#66

Post by [deleted] »

kgeist wrote: post #16

For my experiments with new self-hostable models on Linux, I've been using a script to download GGUF-models from TheBloke on HuggingFace (currently, TheBloke's repository has 657 models in the GGUF format) which I feed to a simple program I wrote which invokes llama.cpp compiled with GPU support. The GGUF format and TheBloke are a blessing, because I'm able to check out new models basically on the day of their releas…

[deleted]

Re: LM Studio – Discover, download, and run local LLMs

#68

Post by [deleted] »

chatmasta wrote: post #56

Am I missing something here? I'm on a recent M2 machine. Every model I've downloaded fails to load immediately when trying to load it. Is there some way to get feedback on the reason for failure, like a log file or something? EDIT: The problem is I'm on macOS 13.2 (Ventura). According to a message in Discord, the minimum version for some (most?) models is 13.6.

[deleted]

Re: LM Studio – Discover, download, and run local LLMs

#69

Post by gsuuon »

cloudking wrote: post #60

Is anyone using open source models to actually get work done or solving problems in their software architecture? So far I haven't found anything near the quality of GPT-4.

Zephyr is coherent enough to bounce ideas off of, but I'm eagerly awaiting when open-source models are on par productivity wise with the big providers. I imagine some folks are utilizing codellama 34b somehow, but I haven't been able to effectively.

Re: LM Studio – Discover, download, and run local LLMs

#70

Post by kkielhofner »

mfalcon wrote: post #50
rsolva wrote: post #28

Earlier quoted context omitted.

Ollama does this. I run it in a container on my homelab (Proxmox on a HP EliteDesk SFF G2 800) and 7B models run decently fast on CPU-only. Ollama has a nice API and makes it easy to manage models. Together with ollama-webui, it can replace ChatGPT 3.5 for most tasks. I also use it in VSCode and nvim with plugins, works great! I have been meaning to write a short blog post about my setup...

I've been trying Ollama locally. I've yet to know how it'll behave in a production setting.

Depending on what you mean by "production" you'll probably want to look at "real" serving implementations like HF TGI, vLLM, lmdeploy, Triton Inference Server (tensorrt-llm), etc. There are also more bespoke implementations for things like serving large numbers of LoRA adapters[0].

These are heavily optimized for more efficient memory usage, performance, and responsiveness when serving large numbers of concurrent requests/users in addition to things like model versioning/hot load/reload/etc, Prometheus metrics, things like that.

One major difference is at this level a lot of the more aggressive memory optimization techniques and support for CPU aren't even considered. Generally speaking you get GPTQ and possibly AWQ quantization + their optimizations + CUDA only. Their target users and their use cases are often using A100/H100 and just trying to need fewer of them. Support for lower VRAM cards, older CUDA compute architectures, etc come secondary to that (for the most part).

[0] - https://github.com/S-LoRA/S-LoRA

Post reply on HN