Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Coming of Local LLMs

nickarner.com

Post reply on HN
Print view View original

31–40 of 214 posts

Re: The Coming of Local LLMs

#31

Post by travisjungroth »

tyingq wrote: post #23
yacine_ wrote: post #17

I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make t…

Agree, though plain old Bayesian classifiers have been able to handle some significant portion of that office work for a long time. And not much ever came from it for everyday stuff outside of spam filters. Maybe both the buzz factor and broader applicability means it's more likely to happen this go around?

Ease of use is huge, so is deployment. Even as a software engineer the overhead for a random classification is so much lower.

Re: The Coming of Local LLMs

#32

Post by londons_explore »

layer8 wrote: post #29

It would be nice to be able to run an LLM-driven spamassassin on a VPS for acceptable cost.

I don't think it will help. Actual friends occasionally send me mail that says "test" from a random account. And spammers do too... There is no way to seperate them.

Re: The Coming of Local LLMs

#33

Post by abetlen »

sendfoods wrote: post #22

How realistic is CPU-only inference in the near future?

You can see for yourself (assuming you have the model weights) https://github.com/abetlen/llama-cpp-python

I get around ~140 ms per token running a 13B parameter model on a thinkpad laptop with a 14 core Intel i7-9750 processor. Because it's CPU inference the initial prompt processing takes longer than on GPU so total latency is still higher than I'd like. I'm working on some caching solutions that should make this bareable for things like chat.

Re: The Coming of Local LLMs

#34

Post by layer8 »

londons_explore wrote: post #32
layer8 wrote: post #29

It would be nice to be able to run an LLM-driven spamassassin on a VPS for acceptable cost.

I don't think it will help. Actual friends occasionally send me mail that says "test" from a random account. And spammers do too... There is no way to seperate them.

Personally I haven’t had that problem. I can readily distinguish spam from non-spam by manual review ~99% of the time. If I could train or instruct an LLM to do the same as I do now, I would be happy. My current false-negative rate with Bayesian spamassassin is more like 50%.

Re: The Coming of Local LLMs

#35

Post by abetlen »

travisjungroth wrote: post #25
sendfoods wrote: post #22

How realistic is CPU-only inference in the near future?

It’s in the near past. https://github.com/ggerganov/llama.cpp

Also worth checking out https://github.com/saharNooby/rwkv.cpp which is based on Georgi's library and offers support for the RWKV family of models which are Apache-2.0 licensed.

Re: The Coming of Local LLMs

#38

Post by rubidium »

kbrkbr wrote: post #7
emrah wrote: post #2

It's great they got LLMs running on resource constrained devices but are they any good? Or I should ask, with the limited resources they get, what good are they for?

From my experience with llama.cpp and oobaboogas webui I can say they are amazing, at least on my gaming pc. I’m absolutely astonished at the speed and quality of llama, alpaca, galactica and vicuna (the >10B parameters ones). Make no mistake, it’s for tinkerers that do not expect each prompt to be answered human like. I see them as creativity and thought testing tools, also knowledge exploratory.

Is there a "getting started" guide you'd recommend to a newbie to the space?
Post reply on HN