Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local LLM inference – impressive but too hard to work with

medium.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Local LLM inference – impressive but too hard to work with

#31

Post by manmal »

antirez wrote: post #7

Download the model in background. Serve the client with an LLM vendor API just for the first requests, or even using that same local LLM installed on your own servers (likely cheaper). By doing so, in the long run the inference cost is near-zero and allows to use LLMs in otherwise impossible business models (like freemium).

Personally, I only use locally run models when I absolutely can’t have the prompt/context uploaded to a cloud. For anything else, I just use one of the commercial cloud hosted models. The ones I‘m using are way faster and better in _every_ way except privacy. Eg if you are ok to spend more, you can get blazing fast DeepSeek v3 or R1 via OpenRouter. Or, rather cheap Claude Sonnet via Copilot (pre-release also has Gemini 2.5 Pro btw).

I’ve gotten carried away - I meant to express that using cloud as a fallback for local models is something I absolutely don’t want or need, because privacy is the whole and only point to local models.

Re: Local LLM inference – impressive but too hard to work with

#32

Post by [deleted] »

thot_experiment wrote: post #24
aazo11 wrote: post #12

Earlier quoted context omitted.

By "too hard" I do not mean getting started with them to run inference on a prompt. Ollama especially makes that quite easy. But as an application developer, I feel these platforms are too hard to build around. The main issues being: getting the correct small enough task specific model and how long it takes to download these models for the end user.

I guess it depends on expectations, if your expectation is an CRUD app that opens in 5 seconds, then sure, it's definitely tedious. People do install things though, the companion app for DJI action cameras is 700mb (which is an abomination, but still). Modern games are > 100gb on the high side, downloading 8-16gb of tensors one time is nbd. You mentioned that there are 663 different models of dsr1-7b on huggingface,…

[deleted]

Re: Local LLM inference – impressive but too hard to work with

#33

Post by aazo11 »

byyoung3 wrote: post #21
aazo11 wrote: post #13

Earlier quoted context omitted.

Exactly. Why does this not exist yet?

its an if statement on whether the model has downloaded or not

A better solution would train/finetune the smaller model from the responses of the larger model and only push to the inference to the edge if the smaller model is performant and the hardware specs can handle the workload?

Re: Local LLM inference – impressive but too hard to work with

#34

Post by CharlieRuan »

ijk wrote: post #8

There's two general categories of local inference: - You're running a personal hosted instance. Good for experimentation and personal use; though there's a tradeoff on renting a cloud server. - You want to run LLM inference on client machines (i.e., you aren't directly supervising it while it is running). I'd say that the article is mostly talking about the second one. Doing the first one will get you familiar enough…

Curious what are some examples of "per-player-action API costs" for games?

Re: Local LLM inference – impressive but too hard to work with

#35

Post by ianand »

larodi wrote: post #30

Having done my masters on the topic of grammar-assisted text2sql let me add some additional context here: - first of all local inference can never beat cloud inference for the very simple reason that costs go down with batching. it took me two years to actually understand what batching is - the LLM tensors flowing through transformer layers has a dimension designed specifically for processing data in parallel. so no…

Sounds like an interesting masters thesis. Is your masters thesis available online somewhere?

Re: Local LLM inference – impressive but too hard to work with

#36

Post by ivape »

CharlieRuan wrote: post #34
ijk wrote: post #8

There's two general categories of local inference: - You're running a personal hosted instance. Good for experimentation and personal use; though there's a tradeoff on renting a cloud server. - You want to run LLM inference on client machines (i.e., you aren't directly supervising it while it is running). I'd say that the article is mostly talking about the second one. Doing the first one will get you familiar enough…

Curious what are some examples of "per-player-action API costs" for games?

What if I charge "whales" in games to talk to an anime girl? Maybe I'll only let you talk to her once a day unless you pay me like a kissing booth for every convo. There's going to be some predatory stuff out there, I can see what the GP is talking about with games.

Re: Local LLM inference – impressive but too hard to work with

#37

Post by kevingadd »

CharlieRuan wrote: post #34
ijk wrote: post #8

There's two general categories of local inference: - You're running a personal hosted instance. Good for experimentation and personal use; though there's a tradeoff on renting a cloud server. - You want to run LLM inference on client machines (i.e., you aren't directly supervising it while it is running). I'd say that the article is mostly talking about the second one. Doing the first one will get you familiar enough…

Curious what are some examples of "per-player-action API costs" for games?

For a while basically any mobile or browser freemium game you tried would have progress timers for building things or upgrading things and they'd charge you Actual Money to skip the wait. That's kind of out of fashion now though some games still do it.

Re: Local LLM inference – impressive but too hard to work with

#38

Post by monoid73 »

aazo11 wrote: post #33
byyoung3 wrote: post #21

Earlier quoted context omitted.

its an if statement on whether the model has downloaded or not

A better solution would train/finetune the smaller model from the responses of the larger model and only push to the inference to the edge if the smaller model is performant and the hardware specs can handle the workload?

yeah, that'd b nice, some kind of self-bootstrapping system where you start with a strong cloud model, then fine-tune a smaller local one over time until it’s good enough to take over. tricky part is managing quality drift and deciding when it's 'good enough' without tanking UX. edge hardware's catching up though, so feels more feasible by the day.

Re: Local LLM inference – impressive but too hard to work with

#39

Post by ijk »

CharlieRuan wrote: post #34
ijk wrote: post #8

There's two general categories of local inference: - You're running a personal hosted instance. Good for experimentation and personal use; though there's a tradeoff on renting a cloud server. - You want to run LLM inference on client machines (i.e., you aren't directly supervising it while it is running). I'd say that the article is mostly talking about the second one. Doing the first one will get you familiar enough…

Curious what are some examples of "per-player-action API costs" for games?

Inference using an API costs money. Not a lot of money, per million tokens, but it adds up if you have a lot of tokens...and some of the obvious game uses really chew through the tokens. Like chatting with a character, or having the NPC character make decisions via reasoning model. Can easily make the tokens add up.

Games, on the other hand, are mostly funded via up-front purchase (so you get the money once and then have to keep the servers running) or free to play, which very carefully tracks user acquisition costs versus revenue. Most F2P games make a tiny amount per player; they make up the difference via volume (and whales). So even a handful of queries per day per player can bankrupt you if you have a million players and no way to recoup the inference cost.

Now, you can obviously add a subscription or ongoing charge to offset it, but that's not how the industry is mostly set up at the moment. I expect that the funding model will change, but meanwhile having a model on the edge device is the only currently realistic way to afford adding an LLM to a big single player RPG, for example.

Re: Local LLM inference – impressive but too hard to work with

#40

Post by ijk »

larodi wrote: post #30

Having done my masters on the topic of grammar-assisted text2sql let me add some additional context here: - first of all local inference can never beat cloud inference for the very simple reason that costs go down with batching. it took me two years to actually understand what batching is - the LLM tensors flowing through transformer layers has a dimension designed specifically for processing data in parallel. so no…

There's local applications of parallel processing; your average chatbot wouldn't use it, but a research bot with multiple simultaneous queries will, for example.

Better local beamsearch would be really nice to have, though.

Post reply on HN