Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Local LLM inference – impressive but too hard to work with

medium.com

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Local LLM inference – impressive but too hard to work with

#41

Post by ijk »

larodi wrote: post #30

Having done my masters on the topic of grammar-assisted text2sql let me add some additional context here: - first of all local inference can never beat cloud inference for the very simple reason that costs go down with batching. it took me two years to actually understand what batching is - the LLM tensors flowing through transformer layers has a dimension designed specifically for processing data in parallel. so no…

I do wonder if recursion is particularly hard for LLMs, given that they have a hard limit on how much they can loop for a given token. (Absent beam search, reasoning models, and other trickery.)

Re: Local LLM inference – impressive but too hard to work with

#42

Post by larodi »

ijk wrote: post #41
larodi wrote: post #30

Having done my masters on the topic of grammar-assisted text2sql let me add some additional context here: - first of all local inference can never beat cloud inference for the very simple reason that costs go down with batching. it took me two years to actually understand what batching is - the LLM tensors flowing through transformer layers has a dimension designed specifically for processing data in parallel. so no…

I do wonder if recursion is particularly hard for LLMs, given that they have a hard limit on how much they can loop for a given token. (Absent beam search, reasoning models, and other trickery.)

Given a prolog (not problog, but the non-stochastic one) source is a parametric grammar, we can perhaps* argue the inference on the programming logic level can be unfolded by using a pen and pencil. think L-systems, they are self-similar, and recursively defined. The catch is that the whole sequence gets rewritten on each step. If you can get the LLM to do this as it progresses with generation - you get recursion. Question is whether you can get the LLM rewrite the context window, and my bet would be someone is already working on it.

* I say perhaps, because PROLOG engines normally don't rewrite strings like crazy while doing inference, so my statement may be somewhat off.

Re: Local LLM inference – impressive but too hard to work with

#43

Post by kmeisthax »

manmal wrote: post #29
bionhoward wrote: post #10

LM Studio seems pretty good at making local models easier to use

A less known feature of LM Studio I really like is speculative decoding: https://lmstudio.ai/blog/lmstudio-v0.3.10 Basically you let a very small model speculate on the next few tokens, and the large model then blesses/rejects those predictions. Depending on how well the small model performs, you get massive speedups that way. The small model has to be as close to the big model as possible - I tried this with models…

How exactly does this give a speedup? If you have to wait for the large model to confirm the small model's predictions, wouldn't it always be slower than just running the large model?

Re: Local LLM inference – impressive but too hard to work with

#44

Post by larodi »

ianand wrote: post #35
larodi wrote: post #30

Having done my masters on the topic of grammar-assisted text2sql let me add some additional context here: - first of all local inference can never beat cloud inference for the very simple reason that costs go down with batching. it took me two years to actually understand what batching is - the LLM tensors flowing through transformer layers has a dimension designed specifically for processing data in parallel. so no…

Sounds like an interesting masters thesis. Is your masters thesis available online somewhere?

Well, not sure about the final doc that went to the university, but this is the almost final draft.

https://docs.google.com/document/d/e/2PACX-1vSyWbtX700kYJgqe...

Since its in Cyrillic you should perhaps use a translation service. There are some screens showing results, though as I was really on a tight deadline, and its not a PHD but masters thesis, I decided to not go into in-depth evaluation of the proposed methodology against SPIDER (https://yale-lily.github.io/spider). Even though you can find the simplifed GBNF grammar, also some of the outputs. The grammar, interestingly it benefits/exploits a bug in llama.cpp which allows some sort of recursively-chained rules. Bibliography is in English, but really - there is so much written on the topic, by no means comprehensive.

Sadly no open inference engine (at time of writing) was both good enough in beam search, and grammars, so this whole things needs to perhaps be redone in pytorch.

If I find myself in a position to do this for commercial goals, I'd also explore the possibility of having human-catered SQLs against the particular schema, in order to guide the model better. And then do RAG on the DB for more context. Note: I'm already doing E/R model reduction to the minimal connected graph which includes all entities of particular interest to the present query.

And finally, since you got that far - the real real problem with restricting LLM output with grammars is the tokenization. Because all parsers work reading one char at a time, and tokens are very often few chars, so the parser in a way needs to be able to "lookahead", which it normally does not. I believe OpenAI wrote they realized this also, but I can't really find the article atm.

Re: Local LLM inference – impressive but too hard to work with

#45

Post by Havoc »

kmeisthax wrote: post #43
manmal wrote: post #29

Earlier quoted context omitted.

A less known feature of LM Studio I really like is speculative decoding: https://lmstudio.ai/blog/lmstudio-v0.3.10 Basically you let a very small model speculate on the next few tokens, and the large model then blesses/rejects those predictions. Depending on how well the small model performs, you get massive speedups that way. The small model has to be as close to the big model as possible - I tried this with models…

How exactly does this give a speedup? If you have to wait for the large model to confirm the small model's predictions, wouldn't it always be slower than just running the large model?

Apparently for the bigger model checking a token is faster than generating a fresh one. So if they tiny model gets it right you get a tiny speed bump. Can’t say I fully understand it either why it’s faster to check

Needs a pretty large difference in size to result in a speedup. 0.5 vs 27b is the only ones I’ve seen a speedbump

Re: Local LLM inference – impressive but too hard to work with

#47

Post by segmondy »

the advantage of local LLM is that you literally could find many models that have no cloud equivalent. someone could have made a fine tune to meet your needs. if you can't find a generic model that meets your need, you can get an appropriate size model you can run, build your or get dataset. then train the cloud, then use the model locally.

Re: Local LLM inference – impressive but too hard to work with

#48

Post by K0balt »

ijk wrote: post #39
CharlieRuan wrote: post #34

Earlier quoted context omitted.

Curious what are some examples of "per-player-action API costs" for games?

Inference using an API costs money. Not a lot of money, per million tokens, but it adds up if you have a lot of tokens...and some of the obvious game uses really chew through the tokens. Like chatting with a character, or having the NPC character make decisions via reasoning model. Can easily make the tokens add up. Games, on the other hand, are mostly funded via up-front purchase (so you get the money once and then…

You release the game with a variable in game experience. If the player has two 4090s chugging away, she runs everything locally. If he’s got an rx480, a barebones 1b model or a subscription for the nicer AI NPCs, which can open up AI driven side quests (adding minor content within having to write it). Include a “free” month when you register the game.

Re: Local LLM inference – impressive but too hard to work with

#50

Post by vunderba »

resource_waste wrote: post #18
bionhoward wrote: post #10

LM Studio seems pretty good at making local models easier to use

I'm genuinely afraid its going to do telemetry one day. I'm sure someone is watching their internet traffic, but I don't. I take the risk now, but I ask questions about myself, relationships, conversations, etc... Stuff I don't exactly want Microsoft/ChatGPT to have.

If you want privacy, use a local models and an open-source chat interface such as OpenWeb-UI or Jan. (avoid proprietary systems such as Msty or LM Studio).

https://github.com/janhq/jan

https://github.com/open-webui/open-webui

Post reply on HN