Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Self-Hosted Llama 3.2 with Coolify on My Home Server

geek.sg

Post reply on HN
Print view View original

11–20 of 94 posts

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#11

Post by khafra »

varun_ch wrote: post #4

I’m curious about how good the performance with local LLMs is on ‘outdated’ hardware like the author’s 2060. I have a desktop with a 2070 super that it could be fun to turn into an “AI server” if I had the time…

If you want to set up an AI server for your own use, it's exceedingly easy to install LM Studio and hit the "serve an API" button.

Testing performance this way, I got about 0.5-1.5 tokens per second with an 8GB 4bit quantized model on an old DL360 rack-mount server with 192GB RAM and 2 E5-2670 CPUs. I got about 20-50 tokens per second on my laptop with a mobile RTX 4080.

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#13

Post by xyc »

taosx wrote: post #9

For the people who self-host LLMs at home: what use cases do you have? Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM summarize, generate hierarchical tags, and store in a database. For the notes part at least, I wouldn't want to give them to another provider; even for the bookmarks, I wouldn't be comfortable passing my reading profile to anyone.

llama3.2 1b & 3b is really useful for quick tasks like creating some quick scripts from some text, then pasting them to execute as it's super fast & replaces a lot of temporary automation needs. If you don't feel like invest time into automation, sometimes you can just feed into an LLM.

This is one of the reason why recently I added floating chat to https://recurse.chat/ to quickly access local LLM.

Here's a demo: https://x.com/recursechat/status/1846309980091330815

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#16

Post by taosx »

khafra wrote: post #11
varun_ch wrote: post #4

I’m curious about how good the performance with local LLMs is on ‘outdated’ hardware like the author’s 2060. I have a desktop with a 2070 super that it could be fun to turn into an “AI server” if I had the time…

If you want to set up an AI server for your own use, it's exceedingly easy to install LM Studio and hit the "serve an API" button. Testing performance this way, I got about 0.5-1.5 tokens per second with an 8GB 4bit quantized model on an old DL360 rack-mount server with 192GB RAM and 2 E5-2670 CPUs. I got about 20-50 tokens per second on my laptop with a mobile RTX 4080.

LM studio is so nice, I'm up and running in 5 minutes. ty

Re: I Self-Hosted Llama 3.2 with Coolify on My Home Server

#17

Post by segalord »

taosx wrote: post #9

For the people who self-host LLMs at home: what use cases do you have? Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM summarize, generate hierarchical tags, and store in a database. For the notes part at least, I wouldn't want to give them to another provider; even for the bookmarks, I wouldn't be comfortable passing my reading profile to anyone.

I use it exclusively for users on my personal website to chat with my data. I've given the setup tools to have read access my files and data
Post reply on HN