Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Coming of Local LLMs

nickarner.com

Post reply on HN
Print view View original

121–130 of 214 posts

Re: The Coming of Local LLMs

#121

Post by ftxbro »

txomon wrote: post #114

Out of doubt, which seems to be spreading around the internet. The LLaMa model weights weren't "leaked" AFAIK but rather explicitly given access to to researchers, isn't it right? I know the article goes on to speak about something else, but I'm not sure why this claim that the LLaMa model weights were leaked, as in unintendenly made available is being done.

My understanding is that researchers could ask for access to weights, but then also they were leaked so that anyone could get them without asking. There is another layer, where Facebook seems to accept it on some level (I mean they don't have a choice anymore anyway); they put a cheeky comment in the open pull request instead of closing it.

Re: The Coming of Local LLMs

#122

Post by pwinnski »

rootusrootus wrote: post #48
hackernewds wrote: post #43

Earlier quoted context omitted.

I disagree with the assessment that Apple is behind. Apple is known for executing well and putting their weight behind the things they launch. Like Jack Dorsey would often say "it's not important to be first to market, you can just be best to market". And the world got CashApp. I'm sure however Apple enters the space, it will be fleshed out (vs Bard).

I want to agree, but it's pretty easy to find instances where Apple has dabbled but not delivered best-in-class solutions. Siri. iCloud. Home automation.

Apple has two values in conflict with each other, I think. On the one hand, they want to deliver best-in-class solutions. On the other hand, they have a commitment to user privacy[0] as perhaps only a gay man growing up in the south might value.

Siri should be better! It lost features post-acquisition by Apple, and it seems like user privacy is why.

Home automation is arguable. If you consider a single point of failure on a server somewhere to be bad, Apple's solution is pretty great. Their commitment to zigging where others zagged put them behind, since hardware vendors didn't want to put in powerful (expensive) enough chips to handle the cryptography, but while other companies go out of business, or transmit images and video to external parties, Apple's works reliably and securely.

Still, as with most of my complaints about Apple, it's a trade-off between privacy and functionality, and Apple will seemingly always choose privacy over functionality, even as Google consistently chooses functionality over privacy.

[0] Yes, there are examples of edge cases that suggest a less-than-perfect record. Contrast that with their competitors, for which invading privacy is foundational to the business model.

Re: The Coming of Local LLMs

#125

Post by wing-_-nuts »

whimsicalism wrote: post #106
wing-_-nuts wrote: post #54

Earlier quoted context omitted.

is there a difference in the quality of llm one would be able to train or run on a gpu with 8, 12, 16, all the way up to 24gb? I'm trying to decide whether it's worth while to splurge on a more expensive 4090 vs a 4070 or whatever.

It makes 0 economical sense to buy a GPU to train a model. If you want to train a model, train it on the cloud.

It's more that I'm building a gaming pc this summer, and I can either target 1440p (4070) for 2k or 4k for 5k (4090). If I can do a lot more with a 4090 over a 4070 it might make sense, but I know a lot of cs students use google colab these days, so I may just rely on that.

Re: The Coming of Local LLMs

#126

Post by yyyk »

whimsicalism wrote: post #78
kbrkbr wrote: post #7

Earlier quoted context omitted.

From my experience with llama.cpp and oobaboogas webui I can say they are amazing, at least on my gaming pc. I’m absolutely astonished at the speed and quality of llama, alpaca, galactica and vicuna (the >10B parameters ones). Make no mistake, it’s for tinkerers that do not expect each prompt to be answered human like. I see them as creativity and thought testing tools, also knowledge exploratory.

If they are amazing, then ChatGPT must be God-like in your view. I've been underwhelmed by Alpaca and Alpaca-LORA and LLaMA all at 13B but I have not tried higher params.

30B Alpaca is better. Not anywhere near ChatGPT level, but better than the 13B models.

Re: The Coming of Local LLMs

#128

Post by yyyk »

drdaeman wrote: post #113
syntaxing wrote: post #72

Earlier quoted context omitted.

Super curious how you did this! Doesn’t 30B model require a hefty computer to run locally (assuming you’re tuning a non-quantized version)

Not at all. Even a Raspberry Pi would do - you only need ~6GiB RAM for a 4-bit quantized LLaMA model (though it's gonna be quite slow). A decent modern desktop machine would do just fine, no need for anything extra fancy. What I'm wondering is how they fed the documents, as all those LLMs have limitations on the input sizes.

>~6GiB RAM

That's for the 7B model. The 30B model needs 24GB quantized (or 64GB for the unquantized model).

Re: The Coming of Local LLMs

#129

Post by simonw »

ingenieroariel wrote: post #76
whimsicalism wrote: post #69

It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force? I've posted this before, but it seems like this genre…

They are comparable to the first ChatGPT proof of concept from say early 2022. The reason many of us are excited is because we may be a year or two away from being able to run a ChatGPT if the open source models follow a similar curve.

I assume you mean GPT-3 - ChatGPT didn't launch until November 30th 2022.

Re: The Coming of Local LLMs

#130

Post by ExxKA »

avereveard wrote: post #95
seydor wrote: post #89

Earlier quoted context omitted.

You make it sound as if ChatGPT itself isnt severly limited. Vicuna and newer 13B models are quite close . And the uncensored models have a capability that ChatGPT will never have.

What are the newer models? I am testing them in batteries across complex tasks, so far vicuña is the most flexible but they all choke on reflective instructions (I. E. Knowledge to extract is not in the model but in the user text)

I very interested in this usecase - where should I go on the interwebs?
Post reply on HN