Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Self-Hosted Llama 3.2 with Coolify on My Home Server

geek.sg

Post reply on HN
Print view View original

41–50 of 94 posts

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#41

Post by bambax »

> I decided to explore self-hosting some of my non-critical applications

Self-hosting static or almost-static websites is now really easy with a Cloudflare front. I just closed my account on SmugMug and published my images locally using my NAS; this costs no extra money (is basically free) since the photos were already on the NAS, and the NAS is already powered on 24-7.

The NAS I use is an Asustor so it's not really Linux and you can't install what you want on it, but it has Apache, Python and PHP with Sqlite extension, which is more than enough for basic websites.

Cloudflare free is like magic. Response times are near instantaneous and setup is minimal. You don't even have to configure an SSL certificate locally, it's all handled for you and works for wildcard subdomains.

And of course if one puts a real server behind it, like in the post, anything's possible.

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#42

Post by wokwokwok »

worldsayshi wrote: post #33
TechDebtDevin wrote: post #30

Earlier quoted context omitted.

I keep an 8b running with ollama/openwebui to ask it to format things, summarization, and to generate SQL/simple bash commands and what not.

So 8b is really smart enough to write scripts for you? How often does it fail?

> So 8b is really smart enough to write scripts for you?

Depends on the model, but in general, no.

...but it's fine for simple 1 liner commands like "how do I revert my commit?" or "rename these files to camelcase".

> How often does it fail?

Immediately and constantly if you ask anything hard.

An 8b model is not chat-gpt. The 3B model in the OP post is not chat-gpt.

The capability compared to sonnet/4o is like a potato and a car.

Search for 'LLM Leaderboard' and you can see for yourself. The 8b models do not even rank. They're generally not capable enough to use as a self hosted assistant.

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#43

Post by seungwoolee518 »

thangngoc89 wrote: post #3
seungwoolee518 wrote: post #2

Great post! However, Do I need to Install CUDA toolkit on host? I haven't install CUDA toolkit when I use on Containerized platform (like docker)

You don't need to install CUDA toolkit on host system. Nvidia driver + Nvidia container toolkit would do the job. You could check official instructions at [0] [0] https://docs.nvidia.com/datacenter/cloud-native/container-to...

Thanks, I was bit confused on install a CUDA Toolkit on the Host. (Because I don't install any software except Driver && Toolkit)

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#44

Post by Reubend »

bambax wrote: post #41

> I decided to explore self-hosting some of my non-critical applications Self-hosting static or almost-static websites is now really easy with a Cloudflare front. I just closed my account on SmugMug and published my images locally using my NAS; this costs no extra money (is basically free) since the photos were already on the NAS, and the NAS is already powered on 24-7. The NAS I use is an Asustor so it's not really…

Is the NAS exposed to the whole internet? Or did you find a clever way to get CloudFlare in front of it despite it just being local?

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#45

Post by afro88 »

xyc wrote: post #13
taosx wrote: post #9

For the people who self-host LLMs at home: what use cases do you have? Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM summarize, generate hierarchical tags, and store in a database. For the notes part at least, I wouldn't want to give them to another provider; even for the bookmarks, I wouldn't be comfortable passing my reading profile to anyone.

llama3.2 1b & 3b is really useful for quick tasks like creating some quick scripts from some text, then pasting them to execute as it's super fast & replaces a lot of temporary automation needs. If you don't feel like invest time into automation, sometimes you can just feed into an LLM. This is one of the reason why recently I added floating chat to https://recurse.chat/ to quickly access local LLM. Here's a demo: ht…

Can you list some real temporary automation needs you've fulfilled? The demo shows asking for facts about space. Lower param models seem to be not great as raw chat models, so I'm interested in what they are doing well for you in this context

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#47

Post by ragebol »

szundi wrote: post #37
ragebol wrote: post #20

Probably saves a bit on the gas bill for heating too

If only we had heat-pump computers

I'd gladly run whatever model you want at home, rent it out so you can pay for both heating, the GPU and the power consumed :-)

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#48

Post by cheema33 »

Reubend wrote: post #44
bambax wrote: post #41

> I decided to explore self-hosting some of my non-critical applications Self-hosting static or almost-static websites is now really easy with a Cloudflare front. I just closed my account on SmugMug and published my images locally using my NAS; this costs no extra money (is basically free) since the photos were already on the NAS, and the NAS is already powered on 24-7. The NAS I use is an Asustor so it's not really…

Is the NAS exposed to the whole internet? Or did you find a clever way to get CloudFlare in front of it despite it just being local?

You can use CloudFlare Tunnel (https://www.cloudflare.com/products/tunnel/) to connect a system to your cloudflare gateway, without exposing it to the Internet.

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#49

Post by thrdbndndn »

dtquad wrote: post #23
netdevnet wrote: post #18

Am I right thinking that a self-hosted llama wouldn't have the kind restrictions ChatGPT has since it has no initial system prompt?

All the self-hosted LLM and text-to-image models come with some restrictions trained into them [1]. However there are plenty of people who have made uncensored "forks" of these models where the restrictions have been "trained away" (mostly by fine-tuning). You can find plenty of uncensored LLM models here: https://ollama.com/library [1]: I personally suspect that many LLMs are still trained on WebText, derivatives of…

My to-go test for uncensoring is to ask the LLM to write erotic novel.

But I haven't yet find any "uncensored" ones (on ollama) that works. Did I miss something?

(On the contrary: when ChatGPT first came out, it was trivial to jailbreak it to make it write erotica.)

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#50

Post by archerx »

bambax wrote: post #41

> I decided to explore self-hosting some of my non-critical applications Self-hosting static or almost-static websites is now really easy with a Cloudflare front. I just closed my account on SmugMug and published my images locally using my NAS; this costs no extra money (is basically free) since the photos were already on the NAS, and the NAS is already powered on 24-7. The NAS I use is an Asustor so it's not really…

You could also use openVPN or wireguard and not have a man in the middle for no reason.

I have a VPN on a raspberry pi and with that I can connect to my self hosted cloud, dev/staging servers for projects, gitlab and etc when I’m not on my home network.

Post reply on HN