How is coolify different than ollama? is it better? worse? I like ollama because I can pull models and it exposes a rest api to me. which is great for development
Might want to skim the article
I Self-Hosted Llama 3.2 with Coolify on My Home Server
71–80 of 94 posts
Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server
#72Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server
#73> I decided to explore self-hosting some of my non-critical applications Self-hosting static or almost-static websites is now really easy with a Cloudflare front. I just closed my account on SmugMug and published my images locally using my NAS; this costs no extra money (is basically free) since the photos were already on the NAS, and the NAS is already powered on 24-7. The NAS I use is an Asustor so it's not really…
Is the NAS exposed to the whole internet? Or did you find a clever way to get CloudFlare in front of it despite it just being local?
The web server of the nas is exposed to the Internet (port forwarding of 80 from the router to the nas); the rest of the nas is not exposed / not accessible from outside the LAN.
The images that are published are low-res versions copied to a directory on a partition accessible to the web server.
This is not the safest solution, as it does punch a hole in the lan... It's kind of an experiment... We'll see how it goes.
Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server
#74For the people who self-host LLMs at home: what use cases do you have? Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM summarize, generate hierarchical tags, and store in a database. For the notes part at least, I wouldn't want to give them to another provider; even for the bookmarks, I wouldn't be comfortable passing my reading profile to anyone.
I work with a lot of attorney’s eyes only documents and most protective orders do not allow for shipping off these files to a third-party.
Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server
#75> I decided to explore self-hosting some of my non-critical applications Self-hosting static or almost-static websites is now really easy with a Cloudflare front. I just closed my account on SmugMug and published my images locally using my NAS; this costs no extra money (is basically free) since the photos were already on the NAS, and the NAS is already powered on 24-7. The NAS I use is an Asustor so it's not really…
You could also use openVPN or wireguard and not have a man in the middle for no reason. I have a VPN on a raspberry pi and with that I can connect to my self hosted cloud, dev/staging servers for projects, gitlab and etc when I’m not on my home network.
I believe the suggested setup was for making a site and images available to the public, for which hiding the origin behind Cloudflare seems a very good reason. Some public IP will need to have ports 443/80 open.
Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server
#76For the people who self-host LLMs at home: what use cases do you have? Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM summarize, generate hierarchical tags, and store in a database. For the notes part at least, I wouldn't want to give them to another provider; even for the bookmarks, I wouldn't be comfortable passing my reading profile to anyone.
I essentially use it like everyone else. I use it to search through my personal documents because I can control the token size and file embedding
Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server
#77Instead of `watch -n 0.5 nvidia-smi` to track GPU usage. One can use `nvtop`
Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server
#78Earlier quoted context omitted.
Or Tailscale, which is pretty cool piece of tech.
Tailscale is wireguard with advertising, a convenient UI, and a STUN/TURN server.
exactly, which means setting up a vps, generating certificates, setting up some type of monitoring to make sure the tunnel is working, etc. I agree that wireguard is the best option, if you have the time and knowledge, but for some dev people that just wants to put up a webpage with a few users, tailscale/cloudflare is a much easier system to maintain (especially as it handles ssl for you as well - to some degree...).
Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server
#79Can you use a selfhosted LLM that fits in 12 GB VRAM as a reasonable substitute for copilot in VSCode? And if so, can you give it documentation and other code repositories to make it better at a particular language and platform?
Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server
#80[deleted]