I Self-Hosted Llama 3.2 with Coolify on My Home Server • Hacker News Board

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#91

Post by xyc » Wed, Oct 16, 2024, 8:23 PM UTC

afro88 wrote: post #45

xyc wrote: post #13

Earlier quoted context omitted.

llama3.2 1b & 3b is really useful for quick tasks like creating some quick scripts from some text, then pasting them to execute as it's super fast & replaces a lot of temporary automation needs. If you don't feel like invest time into automation, sometimes you can just feed into an LLM. This is one of the reason why recently I added floating chat to https://recurse.chat/ to quickly access local LLM. Here's a demo: ht…

Can you list some real temporary automation needs you've fulfilled? The demo shows asking for facts about space. Lower param models seem to be not great as raw chat models, so I'm interested in what they are doing well for you in this context

Things like grab some markdown text and ask to make a pip/npm install one liner, or quick js scripts to paste in the console (which I didn't bother to open an editor), a fun use case was random drawing some lucky winners for the app giveaway from reddit usernames. Mostly it's converting unstructured text to short/one-liner executable scripts & doesn't require much intelligence. For more complex automation/scripts that I'll save for later, I do resort to providers (cursor w sonnet 3.5 mostly).

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#92

Post by theodric » Wed, Oct 16, 2024, 10:37 PM UTC

taosx wrote: post #9

For the people who self-host LLMs at home: what use cases do you have? Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM summarize, generate hierarchical tags, and store in a database. For the notes part at least, I wouldn't want to give them to another provider; even for the bookmarks, I wouldn't be comfortable passing my reading profile to anyone.

I've been enjoying fine-tuning various models with various data, for example 17 years of my own tweets, and then just cranking up the temperature and letting the model generate random crap that cracks me up. Is that practical? Is joy practical? I think there's a place for it.

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#93

Post by m0wer » Sat, Oct 19, 2024, 7:55 PM UTC

taosx wrote: post #9

For the people who self-host LLMs at home: what use cases do you have? Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM summarize, generate hierarchical tags, and store in a database. For the notes part at least, I wouldn't want to give them to another provider; even for the bookmarks, I wouldn't be comfortable passing my reading profile to anyone.

TabbyML! Autocompletion like GitHub Copilot, using qwen-2.5-coder 7B.

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#94

Post by wokwokwok » Mon, Oct 21, 2024, 12:17 AM UTC

lolinder wrote: post #87

wokwokwok wrote: post #42

Earlier quoted context omitted.

> So 8b is really smart enough to write scripts for you? Depends on the model, but in general, no. ...but it's fine for simple 1 liner commands like "how do I revert my commit?" or "rename these files to camelcase". > How often does it fail? Immediately and constantly if you ask anything hard. An 8b model is not chat-gpt. The 3B model in the OP post is not chat-gpt. The capability compared to sonnet/4o is like a pota…

> Search for 'LLM Leaderboard' and you can see for yourself. The 8b models do not even rank. This is not true. On benchmarks, maybe, but I find the LLM Arena more accurately accounts for the subjective experience of using these things, and Llama 3.1 8B ranks relatively high, outperforming GPT-3.5 and certain iterations of 4. Where the 8Bs do struggle is that they don't have as deep a repository of knowledge, so using…

All I can say is my experience is that this is the difference between wanting something to be true, and it actually being true.

> 8B models are extremely fast and cheap to run

yes.

> Combined with good RAG they can do very well.

This is simply not true. They perform at a level which is useful for simple, trivial tasks.

If you consider that 'doing well', then sure.

However, if, like the parent post, you want to be writing scripts, which is specifically what they asked... then: heck, what 8B are you using, because llama 3.1 is shit at it out of the box.

¯\_(ツ)_/¯

A working unit test can take 6 or 7 iterations with a good prompt. Forget writing logic. Creating classes? Using RAG to execute functions from a spec? Forget it.

That's not not the level that I need for an assistant.