Local LLM inference – impressive but too hard to work with
1–10 of 63 posts
Re: Local LLM inference – impressive but too hard to work with
#2I spent a couple of weeks trying out local inference solutions for a project. Wrote up my thoughts with some performance benchmarks in a blog.
TLDR -- What these frameworks can do on off the shelf laptops is astounding. However, it is very difficult to find and deploy a task specific model and the models themselves (even with quantization) are so large the download would kill UX for most applications.
Re: Local LLM inference – impressive but too hard to work with
#3I thought llamafile was supposed to be the solution to "too hard to work with"?
Re: Local LLM inference – impressive but too hard to work with
#4I spent a couple of weeks trying out local inference solutions for a project. Wrote up my thoughts with some performance benchmarks in a blog. TLDR -- What these frameworks can do on off the shelf laptops is astounding. However, it is very difficult to find and deploy a task specific model and the models themselves (even with quantization) are so large the download would kill UX for most applications.
There are ways to improve the performance of local LLMs with inference time techniques. You can try with optillm - https://github.com/codelion/optillm it is possible to match the performance of larger models on narrow tasks by doing more at inference.
Re: Local LLM inference – impressive but too hard to work with
#5I thought llamafile was supposed to be the solution to "too hard to work with"? https://github.com/Mozilla-Ocho/llamafile
Llamafile is great and love it. I run all my models using it and it’s super portable, I have tested it on windows and linux, on a powerful PC and SBC. It worked great without too my issues.
It takes about a month for the features from llama.cpp to trickle in. Also figuring the best mix of context length size to vram size to desired speed takes a while before it gets intuitive.
Re: Local LLM inference – impressive but too hard to work with
#6I thought llamafile was supposed to be the solution to "too hard to work with"? https://github.com/Mozilla-Ocho/llamafile
I thought it's "docker model" (and OCI artifacts).
Re: Local LLM inference – impressive but too hard to work with
#7Download the model in background. Serve the client with an LLM vendor API just for the first requests, or even using that same local LLM installed on your own servers (likely cheaper). By doing so, in the long run the inference cost is near-zero and allows to use LLMs in otherwise impossible business models (like freemium).
Re: Local LLM inference – impressive but too hard to work with
#8There's two general categories of local inference:
- You're running a personal hosted instance. Good for experimentation and personal use; though there's a tradeoff on renting a cloud server.
- You want to run LLM inference on client machines (i.e., you aren't directly supervising it while it is running).
I'd say that the article is mostly talking about the second one. Doing the first one will get you familiar enough with the ecosystem to handle some of the issues he ran into when attempting the second (e.g., exactly which model to use). But the second has a bunch of unique constraints--you want things to just work for your users, after all.
I've done in-browser neural network stuff in the past (back when using TensorFlow.js was a reasonable default choice) and based on the way LLM trends are going I'd guess that edge device LLM will be relatively reasonable soon; I'm not quite sure that I'd deploy it in production this month but ask me again in a few.
Relatively tightly constrained applications are going to benefit more than general-purpose chatbots; pick a small model that's relatively good at your task and train it on enough of your data and you can get a 1B or 3B model that has acceptable performance, let alone the 7B ones being discussed here. It absolutely won't replace ChatGPT (though we're getting closer to replacing ChatGPT 3.5 with small models). But if you've got a specific use case that will hold still enough to deploy a model it can definitely give you the edge versus relying on the APIs.
I expect games to be one of the first to try this: per-player-action API costs murder per-user revenue, most of the gaming devices have some form of GPU already, and most games are shipped as apps so bundling a few more GB in there is, if not reasonable, at least not unprecedented.
Re: Local LLM inference – impressive but too hard to work with
#9Yikes what's the bar for dead simple these days? Even my totally non-technical gamer friends are messing around with ollama because I just have to give them one command to get any of the popular LLMs up and running.
Now of course "non technical" here is still a pc gamer that's had to fix drivers once or twice and messaged me to ask "hey how do i into LLM, Mr. AI knower", but I don't think twice these days about showing any pc owner how to use ollama because I know I probably won't be on the hook for much technical support. My sysadmin friends are easily writing clever scripts against ollama's JSON output to do log analysis and other stuff.
Re: Local LLM inference – impressive but too hard to work with
#10LM Studio seems pretty good at making local models easier to use