Local LLMs remove the data protection problem but open the door for malicious use on a larger scale.
The Coming of Local LLMs
21–30 of 214 posts
Re: The Coming of Local LLMs
#22How realistic is CPU-only inference in the near future?
Re: The Coming of Local LLMs
#23I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make t…
Agree, though plain old Bayesian classifiers have been able to handle some significant portion of that office work for a long time. And not much ever came from it for everyday stuff outside of spam filters.
Maybe both the buzz factor and broader applicability means it's more likely to happen this go around?
Re: The Coming of Local LLMs
#24I expect we will see the biggest jump in performance if (when) consumer-grade coprocessors like mobile GPUs start incorporating attention layers as a primitive building block at the hardware level, e.g., with instructions and memory layouts engineered specifically to make ultra-low-precision (say, 4-bit) transformer layers as compute- and memory-efficient as possible on consumer devices. That seems almost inevitable…
maybe one could use physics to do that in analog. Or even better, in biology. I think with a clump of ~1.5kg of neurons we can have a pretty efficient coprocessor that is fed with pizza.
Re: The Coming of Local LLMs
#25How realistic is CPU-only inference in the near future?
It’s in the near past. https://github.com/ggerganov/llama.cpp
Re: The Coming of Local LLMs
#26I found this to be very liberating, that I can finally type whatever I want into the LLM, without the possibility of the government knowing what I am writing. Just being able to do that, and have the watchful eye of the state not being able to monitor you is amazing.
you have to check your screen's firmware for that
Re: The Coming of Local LLMs
#27I could certainly see them being trained/used as a front end for a database(file system, api calls).
Re: The Coming of Local LLMs
#28I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make t…
Agree, though plain old Bayesian classifiers have been able to handle some significant portion of that office work for a long time. And not much ever came from it for everyday stuff outside of spam filters. Maybe both the buzz factor and broader applicability means it's more likely to happen this go around?
More like: the accessibility is what will make it go around. The accessibility is what changes everything! Getting an easy to use interface (instructions) over python changes the accessibility from the denizens of this website to ~anyone with a computer
If you're interested, see this paper that argues that point: https://arxiv.org/abs/2302.06541
Essentially, being label efficient is more important than being compute efficient, because the biggest computing constraint we have is enough humans doing the labeling (and knowing how to work a jupyter notebook), not tensor smashing nvidia cards
Re: The Coming of Local LLMs
#29It would be nice to be able to run an LLM-driven spamassassin on a VPS for acceptable cost.
Re: The Coming of Local LLMs
#30I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make t…
Agree, though plain old Bayesian classifiers have been able to handle some significant portion of that office work for a long time. And not much ever came from it for everyday stuff outside of spam filters. Maybe both the buzz factor and broader applicability means it's more likely to happen this go around?
I think the real benefits will be for those with an ad-hoc task and no programming/scripting ability.
Sure, you and I know how to write a little script to sort a directory of documents into "schoolwork" and "other stuff".
But most people don't have that ability, so giving them that would really help accessibility.