Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the best self hosted/local alternative to GPT-4?

news.ycombinator.com

Post reply on HN
Print view View original

171–180 of 206 posts

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#171

Post by ijk »

"Okay with paying for it" gives you a wide range of options.

Most of the open source stuff people are talking about is things like running a quantized 33B parameter LLaMA model on a 3090. That can be done on consumer hardware, but isn't quite as good at general purpose queries as GPT-4. Depending on your use case and your ability to fine tune it, that might be sufficient for a number of applications. Partcularly if you've got a very specific task.

However, if you're willing to spend, there are bigger models available (e.g. Falcon 40B, LLaMA 65B) that can be run on data server class machines, if you're willing to spend $15-20K.

Will that get you GPT-4 level inference? Probably not (though it is difficult to quantify); will it get you a high-quality model that can be further fine-tuned on your own data? Yes.

For the smaller models, the fine-tunes for various tasks can be fairly effective; in a few more weeks I expect that they'll have continued to improve significantly. There's new capabilities being added every week.

The biggest weakness that's been highlighted in research is that the open source models aren't as good at the wide range of tasks that OpenAI's RLHF has covered; that's partly a data issue and partly a training issue.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#174

Post by 0xbadc0de5 »

I'll +1 the votes for Guanaco and Vicuna running with the Oobabooga text-generation-webui.

With a 4090, you can get ChatGPT 3.5 level results from Guanaco 33B. Vicuna 13B is a solid performer on more resource-constrained systems.

I'd urge the naysayers who tried the OPT and LLaMA models only to give up to note that the the LLM field is moving very quickly - the current set of models are already vastly superior to the LLaMA models from just two months ago. And there is no sign the progress is slowing - in fact, it seems to be accelerating.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#175

Post by JohnFen »

inductive_magic wrote: post #108
icapybara wrote: post #96

Earlier quoted context omitted.

This point is understated. So many people are going around like "I'm building an AI app!" when the reality is, OpenAI built an AI app, you're just designing a front end for it.

OpenAI builds an LLM and an api-interface to that model. The design of abstractions, prompt engineering, custom fine-tunes and software engineering required to ship a valuable application on top of that interface counts as "building an app" in my book.

> required to ship a valuable application on top of that interface counts as "building an app" in my book.

It's certainly building an app. It's not building an AI app, though. It's building a front-end to an existing AI application.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#176

Post by JohnFen »

Zvez wrote: post #121
inductive_magic wrote: post #108

Earlier quoted context omitted.

OpenAI builds an LLM and an api-interface to that model. The design of abstractions, prompt engineering, custom fine-tunes and software engineering required to ship a valuable application on top of that interface counts as "building an app" in my book.

while this is technically correct, difference is that your product won't survive without openai at this point. If you need the model quality openai provides you are stuck and your product can just disappear. Because llm is core building block, irreplaceable one.

Building a product that relies 100% on a single external vendor is taking a huge risk. So many companies have been burned by this in the past that it's amazing anyone doesn't see it as a risky thing.

So I have to believe that the people making these products are intending to make as much cash as possible up front and aren't aiming for a long-term thing.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#177

Post by pmoriarty »

simmanian wrote: post #167
pmoriarty wrote: post #165

Earlier quoted context omitted.

I tend to use Claude primary for creative writing (at which it's better than GPT4 and even Claude+, in my experience), and for explanations, which are more thorough than GPT4 or ChatGPT (GPT 3.5) without any special prompting. Claude should be more well known, imo. ChatGPT/GPT4 gets all the hype, but Claude is really good too... sometimes even better.

I thought Anthropic has Claude and Claude Instant. When you say Claude sometimes feels better than Claude+, do you mean that you feel Claude Instant sometimes gives better result than Claude?

This is what I referred to as Claude: https://poe.com/Claude-instant

And this is Claude+: https://poe.com/Claude%2B

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#178

Post by ubermon »

ngngngng wrote: post #125
icapybara wrote: post #120

Earlier quoted context omitted.

The LLM _is_ the product. Everything else in the stack is window dressing. I don’t think OpenAI’s contribution should be so understated- they built a technology that was considered science fiction just a few years ago. They deserve all the credit for the “AI”.

Good luck building a moat when 95% of your app is just calling the API everyone else has access to.

Moat does not come from compile time but runtime. The company, the operation, and the accumulated data, the brand, the trust and the reputation.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#179

Post by solarmist »

ubermon wrote: post #178
ngngngng wrote: post #125

Earlier quoted context omitted.

Good luck building a moat when 95% of your app is just calling the API everyone else has access to.

Moat does not come from compile time but runtime. The company, the operation, and the accumulated data, the brand, the trust and the reputation.

There are technology moats as well, but for the most part, you're correct. Everything else usually matters much more.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#180

Post by ubermon »

bigmattystyles wrote: post #39
wokwokwok wrote: post #30

There is literally no alternative. You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you. There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require… Too bad. You’re out of luck. Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to relea…

The model is the spice then basically?

haha was thinking the same. The other day openAI's api hiccup gave me a small panic attack
Post reply on HN