Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the best self hosted/local alternative to GPT-4?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 206 posts

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#81

Post by egonschiele »

yankoff wrote: post #43
wokwokwok wrote: post #30

There is literally no alternative. You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you. There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require… Too bad. You’re out of luck. Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to relea…

Is this opinion based on some benchmarking you (or someone else) did?

I believe there are benchmarks, but I can informally second that opinion. I'm building a writing app (chiseleditor.com) and there is nothing as good as the ChatGPT models right now.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#82

Post by jhbadger »

seszett wrote: post #52
_laiq wrote: post #32

Earlier quoted context omitted.

Incredible how people are underestimating ChatGPT or overestimating open-source models. A basic question: How can i join with SQL a column to a string separated with comma GPT-4: PostgreSQL: SELECT STRING_AGG(columnName, ', ') FROM tableName; Guanaco: Here is an example of how you could use the CONCAT function in MySQL to concatenate a string to a column value in a single-line query: SELECT CONCAT('The total price fo…

My test for open models is surprisingly simple. I just ask "What is the capital of France?" and I haven't had a correct answer yet in any model I tried. They often get Paris right at least, but most other details are wrong. Guanaco says: > The current capital of France is Paris. It has been so since 1982 when it replaced the previous one which was Vichy.

What model size? 7B? 13B? 33B? When people say "Guanaco" or "Alpaca" or "Manticore" they are referring to a whole family of models. The really small ones like 7B often are very weird.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#83

Post by meroes »

wokwokwok wrote: post #30

There is literally no alternative. You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you. There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require… Too bad. You’re out of luck. Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to relea…

Techies repeat what they chastise artists for about Adobe

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#84

Post by TradingPlaces »

As people note, you cannot substitute locally for the Azure GPU cloud that GPT-4 runs on. But I believe that will change, and maybe quickly. After years of explosive exponential growth in model size, all of a sudden, small is beautiful.

The precipitating factor is that running large models for research is very expensive, but pales in comparison to putting these things into production. Expenses rise exponentially with model size. Everyone is looking for ways to make the models smaller and run at the edge. I will note that PaLM 2 is smaller than PaLM, the first time I can remember something like that happening. The smallest version of PaLM 2 can run at the edge. Small is beautiful.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#85

Post by wing-_-nuts »

MacsHeadroom wrote: post #65
jonathan-adly wrote: post #25

I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with rea…

You link to Guanaco-33B, but Guanaco-65B is much more capable. CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF 4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQ

It irritates me to no end that people don't list the system requirements of various models.

How much ram and vram does one need to run 4,13,33,65B models at a reasonable speed?

edit: instead I'll ask this, what's the best model to run on a system with a 24gb 4090 and 64gb of ram?

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#86

Post by airgapstopgap »

logicchains wrote: post #63
airgapstopgap wrote: post #38

There really do not exist any alternatives, self-hosted or not. But more importantly, there may never be, what with the rising tide of AI risks and regulations discourse. It seems that soon training and opensourcing or otherwise making accessible a model of that class will be impossible, even as the cost of its production falls.

The UAE's apparently planning to release a Falcon 180B, which would be the most powerful model you can run at home by far. I don't think they'll bow down to western pressure to lobotomise the models given the country's whole business model is bypassing sanctions and the like.

That's assuming it's substantially superior to stuff like BLOOM. I sure hope that's the correct assumption to make.

Of course, running a 180B dense transformer at home for personal use is utterly impractical.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#88

Post by [deleted] »

_laiq wrote: post #32
jonathan-adly wrote: post #25

I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with rea…

Incredible how people are underestimating ChatGPT or overestimating open-source models. A basic question: How can i join with SQL a column to a string separated with comma GPT-4: PostgreSQL: SELECT STRING_AGG(columnName, ', ') FROM tableName; Guanaco: Here is an example of how you could use the CONCAT function in MySQL to concatenate a string to a column value in a single-line query: SELECT CONCAT('The total price fo…

[deleted]

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#89

Post by sroussey »

rhacker wrote: post #56
wokwokwok wrote: post #30

There is literally no alternative. You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you. There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require… Too bad. You’re out of luck. Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to relea…

Guess there really is a moat.

Company moats can grow and later shrink.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#90

Post by nirav72 »

y-curious wrote: post #51
bigmattystyles wrote: post #39

Earlier quoted context omitted.

The model is the spice then basically?

Fitting analogy. The Kenyan data transcribers that OpenAI used are the Fremen, and transcripts of erotic Harry Potter fanfiction are the Shai-Hulud.

Are the kenyan's really going to rise up like the Freman? Most likely not. If anything, they'll be used for cheap labor when the next new thing comes along.
Post reply on HN