Ask HN: What's the best self hosted/local alternative to GPT-4? • Hacker News Board

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#41

Post by Taek » Wed, May 31, 2023, 2:17 PM UTC

amilios wrote: post #3

How much GPU memory do you have access to? If you can run it, Guanaco-65B is probably as close as you can get in terms of something publicly available. https://github.com/artidoro/qlora . But as other comments mention, it's still noticeably worse in my experience.

Another vote from me for guanaco 65b. Here's a link to the model and weights: https://huggingface.co/TheBloke/guanaco-65B-GPTQ

I use it all the time at home. It's decent at things like summaries and writing content. It follows instructions as well as GPT and isn't nearly as pretentious but it's still a bit pretentious.

It's not very good at code.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#42

Post by MacsHeadroom » Wed, May 31, 2023, 2:17 PM UTC

Guanaco-65B[0] using Basaran[1] for your OpenAI compatible API.

(You can use any ChatGPT front-end which lets you change the OpenAI endpoint URL.)

[0] https://huggingface.co/TheBloke/guanaco-65B-HF A QLoRA finetune of LLaMA-65B by Tim Dettmers from the paper here: https://arxiv.org/abs/2305.14314

[1] https://github.com/hyperonym/basaran

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#43

Post by yankoff » Wed, May 31, 2023, 2:17 PM UTC

wokwokwok wrote: post #30

There is literally no alternative. You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you. There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require… Too bad. You’re out of luck. Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to relea…

Is this opinion based on some benchmarking you (or someone else) did?

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#44

Post by redox99 » Wed, May 31, 2023, 2:18 PM UTC

segmondy wrote: post #22

danpalmer wrote: post #12

> Preferably self-hosted (I'm okay with paying for it) The big models, if even available, need >100GB of graphics memory to run and would likely take minutes to warm up. The pricing available via OpenAI/GCP/etc is only effective when you can multi-tenant many users. The cost to run one of these systems for private use would be ~$250k per year.

... strange. I'm running 30B models on a 10yr old PC with a $400 RTX 3060. Folks can run the 65B models with 4090 or dual 3090. Usually for about a cost of $2500.

How? 24GB VRAM can only handle 30B 4bit. Are you offloading some layers to the CPU? That kills performance AFAIK.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#45

Post by jlund-molfese » Wed, May 31, 2023, 2:18 PM UTC

londons_explore wrote: post #31

jonathan-adly wrote: post #25

I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with rea…

Table 1 in your link ( https://arxiv.org/pdf/2305.14314.pdf ) is a good way to compare models. I'd really like someone to make a big leaderboard/ranking engine which pits all these engines against eachother and publishes the resulting Elo score.

Like https://lmsys.org/blog/2023-05-03-arena/ ?

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#46

Post by brookst » Wed, May 31, 2023, 2:19 PM UTC

Solvency wrote: post #29

AndroTux wrote: post #19

https://gpt4all.io/ works fairly well on my 16 GB M1 Pro MacBook. It's certainly not on a level with ChatGPT, but what is? It's a simple app download and allows you to select from multiple available models. No hacking required.

Why hasn't the community created a distributed Folding@home style GPT4/LLM software, where anyone can ask a question, and all participants machine's are contributing to the computation of the answer? It seems like the ideal way for the open source community. Is it not possible for some reason with the way LLMs work and compute? Or has it simply just not been done yet?

There's a bunch of stuff on this area. See https://www.fedai.org/ for one.

I also remember something more directly analogous to @home, but I'm having a hard time finding it.

Edit: Petals! Thank you sibling commenter!

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#47

Post by anotheryou » Wed, May 31, 2023, 2:20 PM UTC

Here you can try vicunia (and quite a few others) easily https://chat.lmsys.org/

They also have A/B testing with a leaderboard where vicunia wins for the self-hostable ones: https://chat.lmsys.org/?leaderboard

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#48

Post by meroes » Wed, May 31, 2023, 2:22 PM UTC

This is like an artist getting used to Adobe’s products before they’re put behind a wall. And borrowing HN’s attitude to that, you apparently deserve it

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#49

Post by maffydub » Wed, May 31, 2023, 2:23 PM UTC

Solvency wrote: post #29

AndroTux wrote: post #19

https://gpt4all.io/ works fairly well on my 16 GB M1 Pro MacBook. It's certainly not on a level with ChatGPT, but what is? It's a simple app download and allows you to select from multiple available models. No hacking required.

Why hasn't the community created a distributed Folding@home style GPT4/LLM software, where anyone can ask a question, and all participants machine's are contributing to the computation of the answer? It seems like the ideal way for the open source community. Is it not possible for some reason with the way LLMs work and compute? Or has it simply just not been done yet?

Interesting thought!

I think the problem is that it's hard to parallelize this effectively across a network with Internet-scale latency - individual matrix multiplications parallelize very well, but there would need to be coordination of each result, which would be much slower than just doing the computation locally.

In the case of Folding@home, evaluating possible folds could be done completely in parallel and only need to be coordinated on discovery of a plausible fold (which is rare), so distributing over a high-latency network is still beneficial.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#50

Post by yread » Wed, May 31, 2023, 2:23 PM UTC

_laiq wrote: post #32

jonathan-adly wrote: post #25

I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with rea…

Incredible how people are underestimating ChatGPT or overestimating open-source models. A basic question: How can i join with SQL a column to a string separated with comma GPT-4: PostgreSQL: SELECT STRING_AGG(columnName, ', ') FROM tableName; Guanaco: Here is an example of how you could use the CONCAT function in MySQL to concatenate a string to a column value in a single-line query: SELECT CONCAT('The total price fo…

I also prefer the first but your point would be stronger if the example from Guanaco was actually wrong or internally inconsistent