Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Coming of Local LLMs

nickarner.com

Post reply on HN
Print view View original

71–80 of 214 posts

Re: The Coming of Local LLMs

#71

Post by muyuu »

rcme wrote: post #45
yacine_ wrote: post #17

I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make t…

LLMs as general purpose classifiers is a really big deal, especially because you can give them fuzzy instructions. I know people are worried about LLMs and spam, but I think LLMs may provide an opportunity to elevate online discourse by being more efficient at filtering out spam and low quality commentary.

spam and especially phishing will become much, much better

it will be hard to trust anything at all

Re: The Coming of Local LLMs

#72

Post by syntaxing »

yacine_ wrote: post #17

I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make t…

Super curious how you did this! Doesn’t 30B model require a hefty computer to run locally (assuming you’re tuning a non-quantized version)

Re: The Coming of Local LLMs

#73

Post by 1827162 »

binkHN wrote: post #11

This is wonderful. As hardware and software continues to improve, everything seems to find a way to run on ever smaller devices. Guess your own pocket-AGI is not too far away after all.

By the way, I was thinking of something along the lines of a powerful FPGA with direct access to large quantities of very fast NAND flash, likely many chips in parallel, which will save having to load the model into RAM..... So it will be able to directly run from NAND flash, which opens up the possibility of using very large models???

Power consumption would not be an issue if it's used sporadically throughout the day, it's not like it needs to run continuously?

There is still the issue of NAND flash read disturb, which I haven't fully looked into yet.

Re: The Coming of Local LLMs

#74

Post by whimsicalism »

homarp wrote: post #62
ctoth wrote: post #50

Earlier quoted context omitted.

Checkout LoRA and Alpaca LoRA and the whole huge group of people who have already figured this out. I think there was another breakthrough (yesterday?) which is a further adaption of LoRA to touch even less parameters at runtime.

DyLoRA https://news.ycombinator.com/item?id=35514228 is what you meant

Disagree that this is a major breakthrough :) but it is probably a marginal improvement.

Re: The Coming of Local LLMs

#75

Post by whimsicalism »

ctoth wrote: post #50
qumpis wrote: post #18

Earlier quoted context omitted.

Inference, even fine-tuning a few layers would be difficult since one needs to use non-quantized model, I'd imagine

Checkout LoRA and Alpaca LoRA and the whole huge group of people who have already figured this out. I think there was another breakthrough (yesterday?) which is a further adaption of LoRA to touch even less parameters at runtime.

They're not training on edge :)

Re: The Coming of Local LLMs

#76

Post by ingenieroariel »

whimsicalism wrote: post #69

It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force? I've posted this before, but it seems like this genre…

They are comparable to the first ChatGPT proof of concept from say early 2022. The reason many of us are excited is because we may be a year or two away from being able to run a ChatGPT if the open source models follow a similar curve.

Re: The Coming of Local LLMs

#77

Post by croes »

colonwqbang wrote: post #55
croes wrote: post #21

Local LLMs remove the data protection problem but open the door for malicious use on a larger scale.

It was never possible to keep this technology secret for any length of time.

I thought the possibility to run these LLMs on everyday hardware would be further on the future and in the beginning more limited to big servers.

But this could be the equivalent of the Low Orbit Ion Canon for phishers and scammers

Re: The Coming of Local LLMs

#78

Post by whimsicalism »

kbrkbr wrote: post #7
emrah wrote: post #2

It's great they got LLMs running on resource constrained devices but are they any good? Or I should ask, with the limited resources they get, what good are they for?

From my experience with llama.cpp and oobaboogas webui I can say they are amazing, at least on my gaming pc. I’m absolutely astonished at the speed and quality of llama, alpaca, galactica and vicuna (the >10B parameters ones). Make no mistake, it’s for tinkerers that do not expect each prompt to be answered human like. I see them as creativity and thought testing tools, also knowledge exploratory.

If they are amazing, then ChatGPT must be God-like in your view.

I've been underwhelmed by Alpaca and Alpaca-LORA and LLaMA all at 13B but I have not tried higher params.

Re: The Coming of Local LLMs

#79

Post by beiller »

ingenieroariel wrote: post #63

In order to run large language models, we should all be buying a fully loaded Mac Studio (128GB of ram, 20 CPU cores, a lot of GPU and Neural cores.) and putting Linux on it to remove the artificial restrictions. Yes, we will be running them soon in low end hardware, but we need to get at least to GPT-3.5-turbo level of inference speed and quality before we try to make it small. I already started. $ neofetch -` x@dec…

Which LLM can run on apple's neural cores / GPU cores? I can only run on plain ol' CPU cores (llama), and it runs fine on my Ryzen CPU for less than half the price of that system. That being said I'm switching from Ubuntu to Arch cause I'm sick of all my packages being way out of date!

Re: The Coming of Local LLMs

#80

Post by xbmcuser »

tyingq wrote: post #23
yacine_ wrote: post #17

I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make t…

Agree, though plain old Bayesian classifiers have been able to handle some significant portion of that office work for a long time. And not much ever came from it for everyday stuff outside of spam filters. Maybe both the buzz factor and broader applicability means it's more likely to happen this go around?

Ease of use is what is going to change everything. Using natural language to ask something and getting an answer is different from what we had before. I knew I could automate a lot of my paperwork with scripts but as I am not a programer I never gave it more than a cursory look. Last Dec while playing around with ChatGPT I was able to get it to write some python scripts that resulted in my spending less than 20-25 min on tasks that I was spending 20-25 hours on. Now could I have written the scripts myself probably but it would have taken me months whereas with chat gpt it took me 2-3 hours to get a working scripts and another 1-2 hours to optimise them.
Post reply on HN