Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Coming of Local LLMs

nickarner.com

Post reply on HN
Print view View original

191–200 of 214 posts

Re: The Coming of Local LLMs

#191

Post by boppo1 »

jrm4 wrote: post #49

As I see these things come out, it feels like there's not a lot of discussion on which hardware (that isn't one of the fancy new Macs?) As in, there might be a lot of graphics cards out there that could be used here? Is it only Nvidia still, is AMD a possibility? Maybe I'm missing something on how the tech works?

30B llama needs a 3090 or 4090. 13B I think you can get away with a 3/4080. If you have 64 gigs of ram and a beefy CPU you can run even 65B, but boy it's slow.

13B is pretty meh, but 30B is great, if not quite Chatgpt. But I can ask it why my highschool geometry teacher was such a cunt and it will happily discuss the matter without reservation. Very therapeutic.

Re: The Coming of Local LLMs

#192

Post by heyoni »

ChatGTP wrote: post #174
gamegoblin wrote: post #58

Earlier quoted context omitted.

I already have a custom browser plugin that calls out to GPT (gpt-3.5-turbo is cheap and good enough for this) to classify and filter out low-effort, overly negative, or intellectually dishonest HN comments. It significantly improves the experience on this site. Bonus points: I had never written a browser plugin, but GPT4 helped me do it in under half an hour.

Some negative, even overly negative comments contain good ideas though so maybe it’s not a great idea?

Your comment has been detected to contain oppositional tones and has thus been filtered out.

You are hereby removed from the discourse.

/s

Re: The Coming of Local LLMs

#193

Post by MacsHeadroom »

whimsicalism wrote: post #103
kbrkbr wrote: post #99

Earlier quoted context omitted.

In my opinion the problem with these is engineering a good prompt. I read of lots of people only getting nonsense or repetitions, and learned a bit from what they shared. These models are not chat bots. Vicuna is more friendly in that regard. But I’m well aware of their limitations also, and I can see how one can be underwhelmed. They are not jacks of all trades

Alpaca & Alpaca-LoRA are literally trained to be chat bots. Their goal is to be instruct-tuned.

Alpaca-LoRA, and all LoRas, are garbage. Alpaca is horrible compared to newer finetunes. Even cleaning the Alpaca dataset and retraining a cleaned Alpaca improves its performance greatly.

But newer finetunes like Vicuna go well beyond that, including hundreds of thousands of real human conversations with GPT-4 ChatGPT in the dataset (unlike Alpaca's fully synthetic dataset).

Vicuna-13B in 16bit is easily comparable to ChatGPT-3.5 in capability. Newer finetunes coming out nearly every day are going beyond chatGPT-3.5 and getting closer and closer to GPT-4 performance.

You don't even have to install anything to validate this for yourself. There's a live web demo of Vicuna-13B right here: https://chat.lmsys.org/ (disable ad blocker if it does not load)

Re: The Coming of Local LLMs

#194

Post by boppo1 »

wing-_-nuts wrote: post #125
whimsicalism wrote: post #106

Earlier quoted context omitted.

It makes 0 economical sense to buy a GPU to train a model. If you want to train a model, train it on the cloud.

It's more that I'm building a gaming pc this summer, and I can either target 1440p (4070) for 2k or 4k for 5k (4090). If I can do a lot more with a 4090 over a 4070 it might make sense, but I know a lot of cs students use google colab these days, so I may just rely on that.

Get a 13900 or 7950X and 64 gb of ram. You can run llama 30B and 65B, slowly but surely. Play with that before buying a gpu. If you really, really see yourself getting into this, then go ahead and get a 3090 or 4090. But otherwise get a cheaper nvidia card and wait for things to develop a little more. You can still play with ML and CUDA but you'll have cash left for when 50X0s drop, and that will probably be right around when this stuff will really be getting hot (if the current plateau doesn't hold).

Llama is basically an auto-complete right now. We're celebrating baby's first steps. It's not really worth the $600-1000 jump up from cards that can run all current games 4k60.

Re: The Coming of Local LLMs

#195

Post by boppo1 »

kbrkbr wrote: post #52
rubidium wrote: post #38

Earlier quoted context omitted.

Is there a "getting started" guide you'd recommend to a newbie to the space?

I started with ggerganov’s llama.cpp GitHub repo, and went from there. But then again, I know some programming, statistics and machine learning, so it may not be for you, I cannot judge that. Models can be found on huggingface.co, and I’d start with eachadea/ggml-vicuna-13b-4bit, but it needs 10G of cpu-ram. It is very friendly to any prompt though. I read on my way (on reddit, when I recall correctly), that there mu…

additional point of reference I don't know shit about ML or stats, and am a very weak programmer. I just know how to install programs on linux and use CLI in a basic fashion. I have had no problem getting llama.cpp going by following ggerganov's readme.

Re: The Coming of Local LLMs

#196

Post by boppo1 »

BulgarianIdiot wrote: post #159
cgearhart wrote: post #6

Earlier quoted context omitted.

Even with smaller models & more optimized hardware, I think edge compute is going to be power-limited first. Batteries today just won’t support constantly running LLMs. But I joked recently that as long as they prove useful then consumers would be willing to swap their iPhone for the old car battery with a phone handle attached.

The first step is for it to be viable for smaller models on desktops. The rest will follow, as hardware catches up. Hybrid analog-hybrid NN hardware is on the horizon, maybe in 3-4 years. This would allow GPT-4 level performance on an iPhone with plausible battery life. The current hardware of course can't pull anything like this yet. But iPhone supports on-device facial recognition, object recognition, dictation and…

Who is making said hardware?

Re: The Coming of Local LLMs

#197

Post by Zuiii »

autoexec wrote: post #188
londons_explore wrote: post #32

Earlier quoted context omitted.

I don't think it will help. Actual friends occasionally send me mail that says "test" from a random account. And spammers do too... There is no way to seperate them.

Don't you do it somehow? Plus a filter doesn't have to be 100% right all the time. Filtering out what is 99% certain to be or not be spam and leaving the human to cover the tiny number of messages that fall into a grey area would still save a ton of time.

Yes, but I rely on my own personal experiences (online and offline) to determine whether an email is real or a scam. Unless that AI filter taps into my memory, it will likely lead to too many false positives and false negatives.

Re: The Coming of Local LLMs

#198

Post by alden5 »

yacine_ wrote: post #17

I was able to run a LLaMa on my personal machine to run some labeling on my documents, as a test of its capabilities. It was instruct tune. 30b parameters 4 example labels, and I had a binary classifier in seconds. Sure, semantic text classifiers were possible for a while, but making it accessible changes everything. Giving anyone who can use a spreadsheet the power of a local LLM (or, basically free LLMs) can make t…

What makes it so much better than normal text classification for me is it doesn't require tons of training data to accurately classify text. using it to parse craigslist posts which i might find interesting showed very promising results although it's fairly slow on my base m1 machine.

Re: The Coming of Local LLMs

#199

Post by duckmysick »

ChatGTP wrote: post #174
gamegoblin wrote: post #58

Earlier quoted context omitted.

I already have a custom browser plugin that calls out to GPT (gpt-3.5-turbo is cheap and good enough for this) to classify and filter out low-effort, overly negative, or intellectually dishonest HN comments. It significantly improves the experience on this site. Bonus points: I had never written a browser plugin, but GPT4 helped me do it in under half an hour.

Some negative, even overly negative comments contain good ideas though so maybe it’s not a great idea?

That's ok. If it's a good idea, somebody else will talk about in a positive way.

Also, unless you are reading every comment in every thread, you are going to miss a few interesting ideas anyway. That's ok too.

Re: The Coming of Local LLMs

#200

Post by MacsHeadroom »

MacsHeadroom wrote: post #193
whimsicalism wrote: post #103

Earlier quoted context omitted.

Alpaca & Alpaca-LoRA are literally trained to be chat bots. Their goal is to be instruct-tuned.

Alpaca-LoRA, and all LoRas, are garbage. Alpaca is horrible compared to newer finetunes. Even cleaning the Alpaca dataset and retraining a cleaned Alpaca improves its performance greatly. But newer finetunes like Vicuna go well beyond that, including hundreds of thousands of real human conversations with GPT-4 ChatGPT in the dataset (unlike Alpaca's fully synthetic dataset). Vicuna-13B in 16bit is easily comparable t…

Also, InstructGPT was released in January 2021 and Vicuna-13B blows it out of the water.
Post reply on HN