[dead]
Ask HN: What's the best self hosted/local alternative to GPT-4?
21–30 of 206 posts
Re: Ask HN: What's the best self hosted/local alternative to GPT-4?
#22> Preferably self-hosted (I'm okay with paying for it) The big models, if even available, need >100GB of graphics memory to run and would likely take minutes to warm up. The pricing available via OpenAI/GCP/etc is only effective when you can multi-tenant many users. The cost to run one of these systems for private use would be ~$250k per year.
... strange. I'm running 30B models on a 10yr old PC with a $400 RTX 3060. Folks can run the 65B models with 4090 or dual 3090. Usually for about a cost of $2500.
Re: Ask HN: What's the best self hosted/local alternative to GPT-4?
#23It depends what you mean by "viable alternatives" and how much money you are prepared to spend on hardware to self-host. As others have mentioned, you can try llama.cpp and LocalAI, but for most ChatGPT-like applications, you won't get anything like as good results. I've found that using GPT-4 via the OpenAI API is somewhat more reliable than ChatGPT, either via the Playground or via a local chat interface like https://github.com/mckaywrigley/chatbot-ui
Re: Ask HN: What's the best self hosted/local alternative to GPT-4?
#24> Preferably self-hosted (I'm okay with paying for it)
I'm the founder of Mirage Studio and we created https://www.mirage-studio.io/private_chatgpt. A privacy-first ChatGPT alternative that can be hosted on-premise or on a leading EU cloud provider.
Re: Ask HN: What's the best self hosted/local alternative to GPT-4?
#25I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA).
A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with reasonable effort.
0. https://arxiv.org/abs/2305.14314 1. https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi 2. https://huggingface.co/timdettmers/guanaco-33b-merged
Re: Ask HN: What's the best self hosted/local alternative to GPT-4?
#26[flagged]
[deleted]
Re: Ask HN: What's the best self hosted/local alternative to GPT-4?
#27> Preferably self-hosted (I'm okay with paying for it) The big models, if even available, need >100GB of graphics memory to run and would likely take minutes to warm up. The pricing available via OpenAI/GCP/etc is only effective when you can multi-tenant many users. The cost to run one of these systems for private use would be ~$250k per year.
... strange. I'm running 30B models on a 10yr old PC with a $400 RTX 3060. Folks can run the 65B models with 4090 or dual 3090. Usually for about a cost of $2500.
30B isn't big anymore. GPT-4 is rumored to have 1T parameters.
Re: Ask HN: What's the best self hosted/local alternative to GPT-4?
#28If you want/need to go cpu only then llama.cpp, and the assorted front ends people are building for it, is looking like a good project: https://github.com/ggerganov/llama.cpp
your statement sounds like it doesn't support GPU, llama supports GPU as well. However llama is just code to run models. OP needs to find a model that works for them, there are lots of models out there.
IMO, the best out right now is https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored... (in llama.cpp ggmlv3 format). A 30B llama fine-tuned on a mix of WizardLM and Vicuna training data. It does conversational interactions pretty well (the wizard) and handles instruction fine too (vicuna).
But it is definitely no GPT3.5 or GPT4 replacement. It will not be good for getting "work" done or helping do tasks. It's for recreation. If you want a GPT3.5 level LLM, something akin to text-davinci-002, you'll need to do the SFT and RLHF fine tuning of the 65B llama yourself. And that's no small task. Neither is running a 65B model (even at 4 bits).
Re: Ask HN: What's the best self hosted/local alternative to GPT-4?
#29https://gpt4all.io/ works fairly well on my 16 GB M1 Pro MacBook. It's certainly not on a level with ChatGPT, but what is? It's a simple app download and allows you to select from multiple available models. No hacking required.
Why hasn't the community created a distributed Folding@home style GPT4/LLM software, where anyone can ask a question, and all participants machine's are contributing to the computation of the answer? It seems like the ideal way for the open source community. Is it not possible for some reason with the way LLMs work and compute? Or has it simply just not been done yet?
Re: Ask HN: What's the best self hosted/local alternative to GPT-4?
#30There is literally no alternative.
You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you.
There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require…
Too bad. You’re out of luck.
Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to release a better version.
That’s it. Right now, there’s no one else letting people have access to their models which are equivalent to gpt-4.