Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Self-Hosted Llama 3.2 with Coolify on My Home Server

geek.sg

Post reply on HN
Print view View original

21–30 of 94 posts

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#21

Post by netdevnet »

segalord wrote: post #17
taosx wrote: post #9

For the people who self-host LLMs at home: what use cases do you have? Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM summarize, generate hierarchical tags, and store in a database. For the notes part at least, I wouldn't want to give them to another provider; even for the bookmarks, I wouldn't be comfortable passing my reading profile to anyone.

I use it exclusively for users on my personal website to chat with my data. I've given the setup tools to have read access my files and data

Is this not something that you can with non-hosted LLMs like ChatGPT? If you expose your data, it should be able to access it iirc

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#22

Post by whitefables »

satvikpendem wrote: post #14

I love Coolify, used to use v3, anyone know how their v4 is going? I thought it was still a beta release from what I saw on GitHub.

I'm using v4 beta in the blog post. Didn't try v3 so there's no point of comparison but I'm loving it so far!

It was so easy to get other non-AI stuffs running!

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#23

Post by dtquad »

netdevnet wrote: post #18

Am I right thinking that a self-hosted llama wouldn't have the kind restrictions ChatGPT has since it has no initial system prompt?

All the self-hosted LLM and text-to-image models come with some restrictions trained into them [1]. However there are plenty of people who have made uncensored "forks" of these models where the restrictions have been "trained away" (mostly by fine-tuning).

You can find plenty of uncensored LLM models here:

https://ollama.com/library

[1]: I personally suspect that many LLMs are still trained on WebText, derivatives of WebText, or using synthetic data generated by LLMs trained on WebText. This might be why they feel so "censored":

>WebText was generated by scraping only pages linked to by Reddit posts that had received at least three upvotes prior to December 2017. The corpus was subsequently cleaned

The implications of so much AI trained on content upvoted by 2015-2017 redditors is not talked about enough.

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#24

Post by exe34 »

netdevnet wrote: post #18

Am I right thinking that a self-hosted llama wouldn't have the kind restrictions ChatGPT has since it has no initial system prompt?

It has a sanitised output. You might want to look for "abliterated" models, where the general performance might drop a bit but the guard-rails have been diminished.

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#25

Post by j12a »

satvikpendem wrote: post #14

I love Coolify, used to use v3, anyone know how their v4 is going? I thought it was still a beta release from what I saw on GitHub.

Coolify is quite nice, have been running some things with the v4 beta.

It reminds a bit of making web sites with a page builder. Easy to install and click around to get something running without thinking too much about it fairly quickly.

Problems are quite similar also, training wheels getting stuck in the woods more easily, hehe.

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#26

Post by taosx »

xyc wrote: post #13
taosx wrote: post #9

For the people who self-host LLMs at home: what use cases do you have? Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM summarize, generate hierarchical tags, and store in a database. For the notes part at least, I wouldn't want to give them to another provider; even for the bookmarks, I wouldn't be comfortable passing my reading profile to anyone.

llama3.2 1b & 3b is really useful for quick tasks like creating some quick scripts from some text, then pasting them to execute as it's super fast & replaces a lot of temporary automation needs. If you don't feel like invest time into automation, sometimes you can just feed into an LLM. This is one of the reason why recently I added floating chat to https://recurse.chat/ to quickly access local LLM. Here's a demo: ht…

Looks very nice, saved it for later. Last week, I worked on implementing always-on speech-to-text functionality for automating tasks. I've made significant progress, achieving decent accuracy, but I imposed some self-imposed constraints to implement certain parts from scratch to deliver a single binary deployable solution, which means I still have work to do (audio processing is new territory for me). However, I'm optimistic about its potential.

That being said, I think the more straightforward approach would be to utilize an existing library like https://github.com/collabora/WhisperLive/ within a Docker container. This way, you can call it via WebSocket and integrate it with my LLM, which could also serve as a nice feature in your product.

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#27

Post by nubinetwork »

netdevnet wrote: post #18

Am I right thinking that a self-hosted llama wouldn't have the kind restrictions ChatGPT has since it has no initial system prompt?

That depends on the frontend, you can supply a system prompt if you want to... whether it follows it to the letter is another problem...

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#28

Post by dtquad »

varun_ch wrote: post #4

I’m curious about how good the performance with local LLMs is on ‘outdated’ hardware like the author’s 2060. I have a desktop with a 2070 super that it could be fun to turn into an “AI server” if I had the time…

I am using an old laptop with a GTX 1060 6 GB VRAM to run a home server with Ubuntu and Ollama. Because of quantization Ollama can run 7B/8B models on an 8 year old laptop GPU with 6 GB VRAM.

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#29

Post by nubinetwork »

dtquad wrote: post #23
netdevnet wrote: post #18

Am I right thinking that a self-hosted llama wouldn't have the kind restrictions ChatGPT has since it has no initial system prompt?

All the self-hosted LLM and text-to-image models come with some restrictions trained into them [1]. However there are plenty of people who have made uncensored "forks" of these models where the restrictions have been "trained away" (mostly by fine-tuning). You can find plenty of uncensored LLM models here: https://ollama.com/library [1]: I personally suspect that many LLMs are still trained on WebText, derivatives of…

> All the self-hosted [...] text-to-image models come with some restrictions trained into them

https://github.com/huggingface/diffusers/issues/3422

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#30

Post by TechDebtDevin »

taosx wrote: post #9

For the people who self-host LLMs at home: what use cases do you have? Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM summarize, generate hierarchical tags, and store in a database. For the notes part at least, I wouldn't want to give them to another provider; even for the bookmarks, I wouldn't be comfortable passing my reading profile to anyone.

I keep an 8b running with ollama/openwebui to ask it to format things, summarization, and to generate SQL/simple bash commands and what not.
Post reply on HN