Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LM Studio – Discover, download, and run local LLMs

lmstudio.ai

Post reply on HN
Print view View original

71–80 of 157 posts

Re: LM Studio – Discover, download, and run local LLMs

#72

Post by Havoc »

bambax wrote: post #26
kgeist wrote: post #16

For my experiments with new self-hostable models on Linux, I've been using a script to download GGUF-models from TheBloke on HuggingFace (currently, TheBloke's repository has 657 models in the GGUF format) which I feed to a simple program I wrote which invokes llama.cpp compiled with GPU support. The GGUF format and TheBloke are a blessing, because I'm able to check out new models basically on the day of their releas…

I'd like to build myself a headless server to run models, that could be queried from various clients locally on my LAN, but am usure where to start and what the hardware requirements would be. Software can always be changed later but I'd rather buy the hardware parts only once. Do you have recommendations about this? or blog posts to get started? What would be a decent hardware configuration?

> usure where to start and what the hardware requirements would be

Have a look at the localllama subreddit

In short though dual 3090 is common, single 4090 or various flavours of M123 macs. Alternatively p40 can be jury-rigged too but research that carefully. In fact anything with more than one gpu is going to require careful research

Re: LM Studio – Discover, download, and run local LLMs

#73

Post by shanusmagnus »

hugovie wrote: post #33

LMStudio is great to run local LLMs, also support OpenAI-compatible API. In the case you need more advance UI/UX, you can use LMStudio with MindMac( https://mindmac.app ), just check this video for details https://www.youtube.com/watch?v=3KcVp5QQ1Ak .

MindMac is the first example I've seen where the UI for working w/ LLMs is not complete and utter horseshit and starts to support workflows that are sensible.

I will buy this with so much enthusiasm if it holds up. Argh, this has been such a pain point.

Re: LM Studio – Discover, download, and run local LLMs

#74

Post by mfalcon »

kkielhofner wrote: post #70
mfalcon wrote: post #50

Earlier quoted context omitted.

I've been trying Ollama locally. I've yet to know how it'll behave in a production setting.

Depending on what you mean by "production" you'll probably want to look at "real" serving implementations like HF TGI, vLLM, lmdeploy, Triton Inference Server (tensorrt-llm), etc. There are also more bespoke implementations for things like serving large numbers of LoRA adapters[0]. These are heavily optimized for more efficient memory usage, performance, and responsiveness when serving large numbers of concurrent req…

Thanks! Really helpful. I've a 3090 at home and my idea is to do some testing on a similar config in the cloud to have an idea of the amount of requests that could be served.

Re: LM Studio – Discover, download, and run local LLMs

#75

Post by rgbrgb »

NoMoreNicksLeft wrote: post #62
rgbrgb wrote: post #59

LMStudio is great, if a bit daunting. If you’re on Mac and want a native open source interface, try out FreeChat https://www.freechat.run

Thanks for the link. I expected it to not let me run this. I have an intel Macbook, was expecting that I'd need Apple Silicon... am I misunderstanding something? I get fairly fast results at the prompt with the default model. How's this thing running with whatever shitty GPU I have in my laptop?

that's the magic of llama.cpp!

I include a universal binary of llama.cpp's server example to do inference. What's your machine? The lowest spec I've heard it running on is a 2017 iMac with 8GB RAM (~5.5 tokens/s). On my m1 with 64GB RAM I get ~30 tokens per second on the default 7B model.

Re: LM Studio – Discover, download, and run local LLMs

#77

Post by kkielhofner »

mfalcon wrote: post #74
kkielhofner wrote: post #70

Earlier quoted context omitted.

Depending on what you mean by "production" you'll probably want to look at "real" serving implementations like HF TGI, vLLM, lmdeploy, Triton Inference Server (tensorrt-llm), etc. There are also more bespoke implementations for things like serving large numbers of LoRA adapters[0]. These are heavily optimized for more efficient memory usage, performance, and responsiveness when serving large numbers of concurrent req…

Thanks! Really helpful. I've a 3090 at home and my idea is to do some testing on a similar config in the cloud to have an idea of the amount of requests that could be served.

The good news is the number of requests and performance is very impressive. For example, on my RTX 4090 from testing many months ago with lmdeploy (it was the first to support AWQ) I was getting roughly 70 tokens/s each across 10 simultaneous sessions with LLama2-13b-Chat - almost 700 tokens/s total. If I were to test again now with all of the impressive stuff that's been added to all of these I'm sure it would only be better (likely dramatically).

The bad news is because "low VRAM cards" like the 24GB RTX 3090 and RTX 4090 aren't really targetted by these frameworks you'll eventually run into "Yeah you're going to need more VRAM for that model/configuration. That's just how it is." as opposed to some of the approaches for local/single session serving that emphasize memory optimization first and tokens/s for a single session next. Often with no consideration or support at all for multiple simultaneous sessions.

It's certainly possible that with time these serving frameworks will deploy more optimizations and strategies for low VRAM cards but if you look at timelines to even implement quantization support (as one example) it's definitely an after-thought and typically only implemented when it aligns with the overall "more tokens for more users across more sessions on the same hardware" goals.

Loading a 70B model on CPU and getting 3 tokens/s (or whatever) is basically seen as an interesting yet completely impractical and irrelevant curiosity to these projects.

In the end "the right tool for the job" always applies.

Re: LM Studio – Discover, download, and run local LLMs

#79

Post by 0xEF »

pentagrama wrote: post #41

Curious about this and I just download it. Want to try uncensored models. I have a question, looking for the most popular "uncensored" model I just find "TheBloke/Luna-AI-Llama2-Uncensored-GGML", but it has 14 files to download between 2 to 7 GB, I just download the first one: https://imgur.com/a/DE2byOB I try the model and it works: https://imgur.com/a/2vtPcui I should download all the 14 files to get better results…

Honest question from someone new to exploring and using these models; why do you need uncensored? What are the use-cases that would call for it?

Again, not questioning your motives or anything, just straight up curious. To use your example, any of us can find bomb building info online fairly easily, and has been a point of social contention since the Anarchist's cookbook. Nobody needs an uncensored LLM for that, of course.

Post reply on HN