81–90 of 94 posts
Can you use a selfhosted LLM that fits in 12 GB VRAM as a reasonable substitute for copilot in VSCode? And if so, can you give it documentation and other code repositories to make it better at a particular language and platform?
Earlier quoted context omitted.
Or Tailscale, which is pretty cool piece of tech.
Tailscale is wireguard with advertising, a convenient UI, and a STUN/TURN server.
Earlier quoted context omitted.
> Cloudflare free is like magic Cloudflare is pretty strict about the Html to media ratio and might suspend or terminate your account if you are serving too many images. I've read far too many horror stories about this on hn only so please make sure what you're doing is allowed by their TOS.
Cloudflare removed that bit from their TOS entirely about a year ago now. Are you citing a more recent source? PS: talking about Cloudflare being snappy when content is being served from a austore nas made me chuckle.
Can you use a selfhosted LLM that fits in 12 GB VRAM as a reasonable substitute for copilot in VSCode? And if so, can you give it documentation and other code repositories to make it better at a particular language and platform?
Technically, yes, but will yield poor results. We did it internally at big corp n+1 and it, frankly, blows. Other than menial tasks, it's good for nothing but a scout badge.
> I decided to explore self-hosting some of my non-critical applications Self-hosting static or almost-static websites is now really easy with a Cloudflare front. I just closed my account on SmugMug and published my images locally using my NAS; this costs no extra money (is basically free) since the photos were already on the NAS, and the NAS is already powered on 24-7. The NAS I use is an Asustor so it's not really…
You could also use openVPN or wireguard and not have a man in the middle for no reason. I have a VPN on a raspberry pi and with that I can connect to my self hosted cloud, dev/staging servers for projects, gitlab and etc when I’m not on my home network.
Earlier quoted context omitted.
So 8b is really smart enough to write scripts for you? How often does it fail?
> So 8b is really smart enough to write scripts for you? Depends on the model, but in general, no. ...but it's fine for simple 1 liner commands like "how do I revert my commit?" or "rename these files to camelcase". > How often does it fail? Immediately and constantly if you ask anything hard. An 8b model is not chat-gpt. The 3B model in the OP post is not chat-gpt. The capability compared to sonnet/4o is like a pota…
This is not true. On benchmarks, maybe, but I find the LLM Arena more accurately accounts for the subjective experience of using these things, and Llama 3.1 8B ranks relatively high, outperforming GPT-3.5 and certain iterations of 4.
Where the 8Bs do struggle is that they don't have as deep a repository of knowledge, so using them without some form of RAG won't get you as good results as using a plain larger model. But frankly I'm not convinced that RAG-free chat is the future anyway, and 8B models are extremely fast and cheap to run. Combined with good RAG they can do very well.
> I decided to explore self-hosting some of my non-critical applications Self-hosting static or almost-static websites is now really easy with a Cloudflare front. I just closed my account on SmugMug and published my images locally using my NAS; this costs no extra money (is basically free) since the photos were already on the NAS, and the NAS is already powered on 24-7. The NAS I use is an Asustor so it's not really…
Is the NAS exposed to the whole internet? Or did you find a clever way to get CloudFlare in front of it despite it just being local?
For the people who self-host LLMs at home: what use cases do you have? Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM summarize, generate hierarchical tags, and store in a database. For the notes part at least, I wouldn't want to give them to another provider; even for the bookmarks, I wouldn't be comfortable passing my reading profile to anyone.
Earlier quoted context omitted.
> So 8b is really smart enough to write scripts for you? Depends on the model, but in general, no. ...but it's fine for simple 1 liner commands like "how do I revert my commit?" or "rename these files to camelcase". > How often does it fail? Immediately and constantly if you ask anything hard. An 8b model is not chat-gpt. The 3B model in the OP post is not chat-gpt. The capability compared to sonnet/4o is like a pota…
I really hope we can get sonnet like performance down to single consumer level GPU someone soon. Maybe the hardware will get there before the models.
But yes, like the guy above said it's really only helpful for one line commands. Like if I forgot some sort flag thats available for a certain type of command. Or random things I don't work with often enough to memorize their little build commands etc. It's not helpful for programming just simple commands.
It also can help with unstructured or messy data to make it more readable, although there's potential to hallucinate if the context is at all large.