Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Coming of Local LLMs

nickarner.com

Post reply on HN
Print view View original

131–140 of 214 posts

Re: The Coming of Local LLMs

#131

Post by artificial »

gamegoblin wrote: post #58
rcme wrote: post #45

Earlier quoted context omitted.

LLMs as general purpose classifiers is a really big deal, especially because you can give them fuzzy instructions. I know people are worried about LLMs and spam, but I think LLMs may provide an opportunity to elevate online discourse by being more efficient at filtering out spam and low quality commentary.

I already have a custom browser plugin that calls out to GPT (gpt-3.5-turbo is cheap and good enough for this) to classify and filter out low-effort, overly negative, or intellectually dishonest HN comments. It significantly improves the experience on this site. Bonus points: I had never written a browser plugin, but GPT4 helped me do it in under half an hour.

This sounds great, would love to see some more about this endeavor.

Re: The Coming of Local LLMs

#132

Post by jstarfish »

drdaeman wrote: post #113
syntaxing wrote: post #72

Earlier quoted context omitted.

Super curious how you did this! Doesn’t 30B model require a hefty computer to run locally (assuming you’re tuning a non-quantized version)

Not at all. Even a Raspberry Pi would do - you only need ~6GiB RAM for a 4-bit quantized LLaMA model (though it's gonna be quite slow). A decent modern desktop machine would do just fine, no need for anything extra fancy. What I'm wondering is how they fed the documents, as all those LLMs have limitations on the input sizes.

I've seen reports that it wrecks RPi SD cards in short order though, so beware...

> What I'm wondering is how they fed the documents, as all those LLMs have limitations on the input sizes.

It's like file hashing at scale, you don't have to read the whole stream for every file, just the first 1024/2048 bytes (or first few paragraphs).

(This works for classification and sorting, less so for summarization.)

Re: The Coming of Local LLMs

#133

Post by la64710 »

One simple thing these LLM models cannot do yet .. that is to simply point a LLM to a URL and it will start scraping - ie follow the hyperlinks and start consuming the content. I am not an AI guy but I guess this has to do with the context limitations of most model? How did they train OpenAI with all internet data till 2021? This I think will be a most popular feature for LLM models and I seriously hope it is OSS whenever it comes out.

Re: The Coming of Local LLMs

#134

Post by metalliqaz »

gamegoblin wrote: post #58
rcme wrote: post #45

Earlier quoted context omitted.

LLMs as general purpose classifiers is a really big deal, especially because you can give them fuzzy instructions. I know people are worried about LLMs and spam, but I think LLMs may provide an opportunity to elevate online discourse by being more efficient at filtering out spam and low quality commentary.

I already have a custom browser plugin that calls out to GPT (gpt-3.5-turbo is cheap and good enough for this) to classify and filter out low-effort, overly negative, or intellectually dishonest HN comments. It significantly improves the experience on this site. Bonus points: I had never written a browser plugin, but GPT4 helped me do it in under half an hour.

How do you define the cost function for 'low effort'?

Re: The Coming of Local LLMs

#135

Post by UncleEntity »

whimsicalism wrote: post #91
seydor wrote: post #89

Earlier quoted context omitted.

You make it sound as if ChatGPT itself isnt severly limited. Vicuna and newer 13B models are quite close . And the uncensored models have a capability that ChatGPT will never have.

They just aren't quite close. Maybe you guys are asking shallow questions, but I'm asking subject matter questions and this is just not true. It would be great if we could have detailed QA evaluation to show this, but of course then the open source people would train their models on it as a fine-tuning datasaet.

Train it on specific subject matter and not waste space on things like “what year did Jack Nicholson beat on that dude’s car with a golf club?”

I mean… the horror.

Re: The Coming of Local LLMs

#137

Post by alchemist1e9 »

simion314 wrote: post #83
whimsicalism wrote: post #69

It appears there is this genre of articles pretending that LLAMA or its RL-HF tuned variants are somehow even close to an alternative to ChatGPT. Spending more than a few moments interacting even with the larger instruct-tuned variants of these models quickly dispels that idea. Why do these takes around open-source AI remain so popular? What is the driving force? I've posted this before, but it seems like this genre…

People are enthusiastic about the possibilities, imagine a black box on your desk contains only RAM and matrix multiplication chips, you install on it your favorite AI assistant and you train it with your private data/code, you remove all prudish restrictions and get productive on your work and on your off times. Llama has the potential to reach ChatGPT it needs tunning to get better at responding to questions, llama…

My hunch is model weights will be commercialized as a purchased object. They can be watermarked so easy to trace any leak.

Then hardware will be a separate business. I think Apple might be caught off-guard by Nvidia on hardware. The latest NVlink and 400Gbps interconnects when combined with H100 next iterations and also rumors of advanced PCIe motherboards with high lane Nvidia CPUs and it looks to me that next year they can be selling $100-300K physical systems optimized for LLM inference that physically remind me of mainframes.

Re: The Coming of Local LLMs

#138

Post by turmeric_root »

txomon wrote: post #114

Out of doubt, which seems to be spreading around the internet. The LLaMa model weights weren't "leaked" AFAIK but rather explicitly given access to to researchers, isn't it right? I know the article goes on to speak about something else, but I'm not sure why this claim that the LLaMa model weights were leaked, as in unintendenly made available is being done.

The model weights were only shared by FB to people who applied for research access. Github repos containing links to the model weights have been taken down by FB.

Re: The Coming of Local LLMs

#139

Post by gamegoblin »

metalliqaz wrote: post #134
gamegoblin wrote: post #58

Earlier quoted context omitted.

I already have a custom browser plugin that calls out to GPT (gpt-3.5-turbo is cheap and good enough for this) to classify and filter out low-effort, overly negative, or intellectually dishonest HN comments. It significantly improves the experience on this site. Bonus points: I had never written a browser plugin, but GPT4 helped me do it in under half an hour.

How do you define the cost function for 'low effort'?

I give GPT some examples of what I consider low-effort comments and non-low-effort comments, and then I just ask it to do classification. It's quite good at few-shot classification of fuzzy stuff like this.

Re: The Coming of Local LLMs

#140

Post by whimsicalism »

UncleEntity wrote: post #135
whimsicalism wrote: post #91

Earlier quoted context omitted.

They just aren't quite close. Maybe you guys are asking shallow questions, but I'm asking subject matter questions and this is just not true. It would be great if we could have detailed QA evaluation to show this, but of course then the open source people would train their models on it as a fine-tuning datasaet.

Train it on specific subject matter and not waste space on things like “what year did Jack Nicholson beat on that dude’s car with a golf club?” I mean… the horror.

That's just not how these models work.
Post reply on HN