LM Studio seems pretty good at making local models easier to use
they made it so easy to do specdec, that alone sold it for me
Some models have even a 0.5B draft model. The speed increase is incredible.
Local LLM inference – impressive but too hard to work with
11–20 of 63 posts
Re: Local LLM inference – impressive but too hard to work with
#12Yikes what's the bar for dead simple these days? Even my totally non-technical gamer friends are messing around with ollama because I just have to give them one command to get any of the popular LLMs up and running. Now of course "non technical" here is still a pc gamer that's had to fix drivers once or twice and messaged me to ask "hey how do i into LLM, Mr. AI knower", but I don't think twice these days about showi…
By "too hard" I do not mean getting started with them to run inference on a prompt. Ollama especially makes that quite easy. But as an application developer, I feel these platforms are too hard to build around. The main issues being: getting the correct small enough task specific model and how long it takes to download these models for the end user.
Re: Local LLM inference – impressive but too hard to work with
#13Download the model in background. Serve the client with an LLM vendor API just for the first requests, or even using that same local LLM installed on your own servers (likely cheaper). By doing so, in the long run the inference cost is near-zero and allows to use LLMs in otherwise impossible business models (like freemium).
Exactly. Why does this not exist yet?
Re: Local LLM inference – impressive but too hard to work with
#14LM Studio seems pretty good at making local models easier to use
They look awesome. Will try it out.
Re: Local LLM inference – impressive but too hard to work with
#15LM Studio seems pretty good at making local models easier to use
Here is another:
Re: Local LLM inference – impressive but too hard to work with
#16There's two general categories of local inference: - You're running a personal hosted instance. Good for experimentation and personal use; though there's a tradeoff on renting a cloud server. - You want to run LLM inference on client machines (i.e., you aren't directly supervising it while it is running). I'd say that the article is mostly talking about the second one. Doing the first one will get you familiar enough…
Very interesting. I had not thought about gaming at all but that makes a lot of sense.
I also agree the goal should not be to replace ChatGPT. I think ChatGPT is way overkill for a lot of the workloads it is handling. A good solution should probably use the cloud LLM outputs to train a smaller model to deploy in the background.
Re: Local LLM inference – impressive but too hard to work with
#17[deleted]
Re: Local LLM inference – impressive but too hard to work with
#18LM Studio seems pretty good at making local models easier to use
I'm genuinely afraid its going to do telemetry one day.
I'm sure someone is watching their internet traffic, but I don't.
I take the risk now, but I ask questions about myself, relationships, conversations, etc... Stuff I don't exactly want Microsoft/ChatGPT to have.
Re: Local LLM inference – impressive but too hard to work with
#19LM Studio seems pretty good at making local models easier to use
I'm genuinely afraid its going to do telemetry one day. I'm sure someone is watching their internet traffic, but I don't. I take the risk now, but I ask questions about myself, relationships, conversations, etc... Stuff I don't exactly want Microsoft/ChatGPT to have.
Local inferencing is synonymous with privacy for me. There is no universe until laws get put into effect where your LLM usage online is private as it stands now. I suspect most of these companies are going to be putting in a Microsoft Clippy style assistant in soon that will act as a recommendation/ad engine very soon, and this of course requires parsing every convo you've ever had. Paid tier may remove Clippy, but boy oh boy the free tier (which most people will use) won't.
Clippy is coming back guys, and we have to be ready for it.
Re: Local LLM inference – impressive but too hard to work with
#20[flagged]