Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the best self hosted/local alternative to GPT-4?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 206 posts

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#61

Post by deet »

In our experimentation, we've found that it really depends what you're looking for. That is you really need to break down down evaluation by task. Local models don't have the power yet to just "do it all well" like GPT4.

There are open source models that are fine tuned for different tasks, and if you're able to pick a specific model for a specific use case you'll get better results.

---

For example, for chat there are models like `mpt-7b-chat` or `GPT4All-13B-snoozy` or `vicuna` that do okay for chat, but are not great at reasoning or code.

Other models are designed for just direct instruction following, but are worse at chat `mpt-7b-instruct`

Meanwhile, there are models designed for code completion like from replit and HuggingFace (`starcoder`) that do decently for programming but not other tasks.

---

For UI the easiest way to get a feel for quality of each of the models (or, chat models at least) is probably https://gpt4all.io/.

And as others have mentioned, for providing an API that's compatible with OpenAI, https://github.com/go-skynet/LocalAI seems to be the frontrunner at the moment.

---

For the project I'm working on (in bio) we're currently struggling with this problem too since we want a nice UI, good performance, and the ability for people to keep their data local.

So at least for the moment, there's no single drop-in replacement for all tasks. But things are changing every week and every day, and I believe that open-source and local can be competitive in the end.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#62

Post by dustypotato »

_laiq wrote: post #32
jonathan-adly wrote: post #25

I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with rea…

Incredible how people are underestimating ChatGPT or overestimating open-source models. A basic question: How can i join with SQL a column to a string separated with comma GPT-4: PostgreSQL: SELECT STRING_AGG(columnName, ', ') FROM tableName; Guanaco: Here is an example of how you could use the CONCAT function in MySQL to concatenate a string to a column value in a single-line query: SELECT CONCAT('The total price fo…

I have chatGPT with GPT-4 model. It never gives such concise answers. Is it because you used the API?

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#63

Post by logicchains »

airgapstopgap wrote: post #38

There really do not exist any alternatives, self-hosted or not. But more importantly, there may never be, what with the rising tide of AI risks and regulations discourse. It seems that soon training and opensourcing or otherwise making accessible a model of that class will be impossible, even as the cost of its production falls.

The UAE's apparently planning to release a Falcon 180B, which would be the most powerful model you can run at home by far. I don't think they'll bow down to western pressure to lobotomise the models given the country's whole business model is bypassing sanctions and the like.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#64

Post by wild_egg »

_laiq wrote: post #32
jonathan-adly wrote: post #25

I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with rea…

Incredible how people are underestimating ChatGPT or overestimating open-source models. A basic question: How can i join with SQL a column to a string separated with comma GPT-4: PostgreSQL: SELECT STRING_AGG(columnName, ', ') FROM tableName; Guanaco: Here is an example of how you could use the CONCAT function in MySQL to concatenate a string to a column value in a single-line query: SELECT CONCAT('The total price fo…

> join with SQL a column to a string

As a human, I wouldn't know what to give you for this either. Not surprised the models give unexpected results. "Guanaco" gives something more in line with what I understand you to be asking for, but clearly you're asking for something else that I don't understand.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#65

Post by MacsHeadroom »

jonathan-adly wrote: post #25

I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with rea…

You link to Guanaco-33B, but Guanaco-65B is much more capable.

CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML

GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF

4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQ

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#66

Post by nickthegreek »

yankoff wrote: post #43
wokwokwok wrote: post #30

There is literally no alternative. You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you. There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require… Too bad. You’re out of luck. Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to relea…

Is this opinion based on some benchmarking you (or someone else) did?

Nothing that you can self host seems to come close to gpt 3.5, let alone gpt-4. r/LocalLlama is good subreddit to lurk to get a pulse on the local llms. Current leader seems to be Guanaco-65B.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#67

Post by captainmuon »

I think you have to distinguish between self-hosted to run on CPU (like LLAMA), on consumer GPU or on big GPUs. I find the market currently very confusing.

I'm especially interested since the data center I'm working for is sitting on a bunch of A100 and I get daily requests of people asking for LLMs tuned to specific cases, who can't or won't use OpenAI for various reasons.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#68

Post by npteljes »

amilios wrote: post #3

How much GPU memory do you have access to? If you can run it, Guanaco-65B is probably as close as you can get in terms of something publicly available. https://github.com/artidoro/qlora . But as other comments mention, it's still noticeably worse in my experience.

I was wondering how much GPU memory does G-65B need, and from the docs, it's "48GB".

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#69

Post by simonw »

seszett wrote: post #52
_laiq wrote: post #32

Earlier quoted context omitted.

Incredible how people are underestimating ChatGPT or overestimating open-source models. A basic question: How can i join with SQL a column to a string separated with comma GPT-4: PostgreSQL: SELECT STRING_AGG(columnName, ', ') FROM tableName; Guanaco: Here is an example of how you could use the CONCAT function in MySQL to concatenate a string to a column value in a single-line query: SELECT CONCAT('The total price fo…

My test for open models is surprisingly simple. I just ask "What is the capital of France?" and I haven't had a correct answer yet in any model I tried. They often get Paris right at least, but most other details are wrong. Guanaco says: > The current capital of France is Paris. It has been so since 1982 when it replaced the previous one which was Vichy.

I just got "The capital of France is Paris." from vicuna-v1-7B running entirely on my iPhone (using the MLC Chat app).

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#70

Post by daryl149 »

Gijs4g wrote: post #24

> Preferably self-hosted (I'm okay with paying for it) I'm the founder of Mirage Studio and we created https://www.mirage-studio.io/private_chatgpt . A privacy-first ChatGPT alternative that can be hosted on-premise or on a leading EU cloud provider.

Haha, beat me to it by a minute! But idd, do check out our self hosted option :D
Post reply on HN