Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Coming of Local LLMs

nickarner.com

Post reply on HN
Print view View original

171–180 of 214 posts

Re: The Coming of Local LLMs

#171

Post by sliken »

turmeric_root wrote: post #97
wing-_-nuts wrote: post #54

Earlier quoted context omitted.

is there a difference in the quality of llm one would be able to train or run on a gpu with 8, 12, 16, all the way up to 24gb? I'm trying to decide whether it's worth while to splurge on a more expensive 4090 vs a 4070 or whatever.

More VRAM => larger models. IME it is absolutely worth maxing out VRAM for the significant improvement in quality, especially with LLaMA (though even with a 4090, you won't be able to run the largest 65-billion parameter model even with 4-bit quantization). That said, I recommend renting a cloud GPU for a few hours and trying the larger models on them before buying a GPU of your own, just to see if the models meet yo…

But should fit easily on a Apple MBP or Studio with 96GB or 128GB of unified memory.

Re: The Coming of Local LLMs

#172

Post by whimsicalism »

throwawaymaths wrote: post #164
whimsicalism wrote: post #101

Earlier quoted context omitted.

They don't

Do you know why/why not?

They've just gone with a different approach for fast attention. [0] Not sure it was due to any particular merits/lack of the nystroformer.

[0]: https://twitter.com/typedfemale/status/1609867110695735296

Re: The Coming of Local LLMs

#173

Post by thewataccount »

whimsicalism wrote: post #170
thewataccount wrote: post #168

Earlier quoted context omitted.

This isn't necessarily true with LoRAs - a 4090 can train/compute the alpaca dataset with LoRA in under 6 hours (it might be 3, I forget what it was). So finetuning with LoRAs and a few other methods is fine on higher end consumer hardware like a 4090 and finishes in a reasonable amount of time - IMO definitely worth it if you're experimenting with this especially for the inference. The base training though yeah I to…

Even with LoRA the economics are not in your favor to buy a 4090 instead of cloud training.

Actually yeah you're definitely right.

My perspective was for people who have other uses for them e.g. gaming or local inference. From a pure finance standpoint you're definitely right - you should rent and not buy a dedicated card. I think you'd need a few thousand hours to break even which is a few months 24/7.

Re: The Coming of Local LLMs

#174

Post by ChatGTP »

gamegoblin wrote: post #58
rcme wrote: post #45

Earlier quoted context omitted.

LLMs as general purpose classifiers is a really big deal, especially because you can give them fuzzy instructions. I know people are worried about LLMs and spam, but I think LLMs may provide an opportunity to elevate online discourse by being more efficient at filtering out spam and low quality commentary.

I already have a custom browser plugin that calls out to GPT (gpt-3.5-turbo is cheap and good enough for this) to classify and filter out low-effort, overly negative, or intellectually dishonest HN comments. It significantly improves the experience on this site. Bonus points: I had never written a browser plugin, but GPT4 helped me do it in under half an hour.

Some negative, even overly negative comments contain good ideas though so maybe it’s not a great idea?

Re: The Coming of Local LLMs

#175

Post by taneq »

whimsicalism wrote: post #140
UncleEntity wrote: post #135

Earlier quoted context omitted.

Train it on specific subject matter and not waste space on things like “what year did Jack Nicholson beat on that dude’s car with a golf club?” I mean… the horror.

That's just not how these models work.

You’ll have to forgive them, they’re probably trained mostly on HN posts. ;)

Re: The Coming of Local LLMs

#176

Post by tough »

darkgreene wrote: post #66
gamegoblin wrote: post #58

Earlier quoted context omitted.

I already have a custom browser plugin that calls out to GPT (gpt-3.5-turbo is cheap and good enough for this) to classify and filter out low-effort, overly negative, or intellectually dishonest HN comments. It significantly improves the experience on this site. Bonus points: I had never written a browser plugin, but GPT4 helped me do it in under half an hour.

Do you have a repo you can share? I'd be very interested in running my own local copy

+1

Re: The Coming of Local LLMs

#177

Post by jpe90 »

gamegoblin wrote: post #166
jpe90 wrote: post #157

Earlier quoted context omitted.

This sounds super cool, I’m very curious how you implemented it. I’m only vaguely familiar with the API.. if I had to guess I would say you send - a system instruction that its job is to filter unwanted content - examples of unwanted content - an instruction like “filter the following html:” For every web request you want to filter, you would re-send all of those messages followed by the page HTML as the final messag…

Close, but it's a bit more specialized to just work on HN: - Examples of unwanted content - Then I give a large numbered list of comments and ask which numbers should be filtered - The plugin then just deletes those comment nodes from the DOM. If HN ever updates their HTML I will have to tweak this code. The reason to send a large list of comments is just to save on costs. It's cheaper to do it this way than one comm…

Oh that’s clever! Thanks for answering!

Re: The Coming of Local LLMs

#178

Post by sliken »

pwinnski wrote: post #122
rootusrootus wrote: post #48

Earlier quoted context omitted.

I want to agree, but it's pretty easy to find instances where Apple has dabbled but not delivered best-in-class solutions. Siri. iCloud. Home automation.

Apple has two values in conflict with each other, I think. On the one hand, they want to deliver best-in-class solutions. On the other hand, they have a commitment to user privacy[0] as perhaps only a gay man growing up in the south might value. Siri should be better! It lost features post-acquisition by Apple, and it seems like user privacy is why. Home automation is arguable. If you consider a single point of failu…

Up until recently I'd have agreed on Apple's.

Increasingly Apple seem to be blocking tracking, noteably from facebook, to make the most profit of that tracking. I've read claims that apple made between $5B and $20B on advertising in 2022. It's far from clear that Apple's view on privacy is going to stay the same.

Re: The Coming of Local LLMs

#179

Post by SV_BubbleTime »

muyuu wrote: post #161
SV_BubbleTime wrote: post #151

Earlier quoted context omitted.

Which will force drive signed communications. So... win.

Yea in a sense it will be a win, but it will up the stakes. A lot of people with get scammed. A lot more than right now.

Disagree. Adding signed communications as the default to replace email's completely nonsense zero encryption model is not upping the stakes. Email is the most successful federated software in the universe, but it's fucking time to get some encryption and authn.

Re: The Coming of Local LLMs

#180

Post by muyuu »

SV_BubbleTime wrote: post #179
muyuu wrote: post #161

Earlier quoted context omitted.

Yea in a sense it will be a win, but it will up the stakes. A lot of people with get scammed. A lot more than right now.

Disagree. Adding signed communications as the default to replace email's completely nonsense zero encryption model is not upping the stakes. Email is the most successful federated software in the universe, but it's fucking time to get some encryption and authn.

you are having a good laugh if you think the average granny is going to do that anytime soon
Post reply on HN