Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the best self hosted/local alternative to GPT-4?

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 206 posts

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#181

Post by cudgy »

JohnFen wrote: post #176
Zvez wrote: post #121

Earlier quoted context omitted.

while this is technically correct, difference is that your product won't survive without openai at this point. If you need the model quality openai provides you are stuck and your product can just disappear. Because llm is core building block, irreplaceable one.

Building a product that relies 100% on a single external vendor is taking a huge risk. So many companies have been burned by this in the past that it's amazing anyone doesn't see it as a risky thing. So I have to believe that the people making these products are intending to make as much cash as possible up front and aren't aiming for a long-term thing.

Use the facade pattern so it’s easier to swap out if needed.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#182

Post by hunter2_ »

amluto wrote: post #100
RC_ITR wrote: post #94

Earlier quoted context omitted.

> You’re faking the reader out by using “join” to mean something which is not JOIN. Yeah and not having to do precise prompt engineering is a major advantage of GPT-4 vs other models.

I don’t think that “prompt engineering” is quite the same thing as clearly specifying the problem. If I’m using a magic AI code-writing companion or asking an actual human to write code for me, I need to define the problem well enough if I want any expectation of getting _correct_ code. A skilled human will distinguish themselves from a less skilled human by also thinking about the overall problem and helping make su…

> helping make sure the specification is correct. This far, I have not seen an AI meet that bar

I've only used Bard so far, but indeed, it has never replied to my question with a clarifying question. Only with a statement. This seems like an obvious evolution of the chatbot concept...

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#183

Post by bzmrgonz »

cloverich wrote: post #133
rhacker wrote: post #122

Earlier quoted context omitted.

So like, would that not suggest that Google has a moat with AI, as it is already intertwined with our email, calendars, etc... (if they ever make their AI public)

It makes it all the more interesting that Google is currently losing to an upstart. They have a MASSIVE advantage in data and moats, but cannot execute on novel products to take advantage of it, despite also hiring a significant chunk of the talent in the market as well. But perhaps at this stage Google's not meant to / need to build cutting edge products, but rather focus on commoditizing the ones that prove profita…

or maybe they like their chinese wall between their internal AI and the public at large!!!

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#184

Post by killingtime74 »

icapybara wrote: post #96
wokwokwok wrote: post #30

There is literally no alternative. You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you. There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require… Too bad. You’re out of luck. Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to relea…

This point is understated. So many people are going around like "I'm building an AI app!" when the reality is, OpenAI built an AI app, you're just designing a front end for it.

Wait so when I'm using postgres I'm just designing a front end for it? Maybe I'm just designing a front end for my CPU?

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#185

Post by JimtheCoder »

RecycledEle wrote: post #92

I often worry about aa "The Machine Stops" scenario. GPT AI actually gives me hope. What if we can store and run an AI in a phone-sized-device that is superior to a similarly sized library of books? Can we have a rugged, solar-powered device that could survive the fall of Civilization and help us rebuild? It would certainly have military applications in a warfare. Imagine being the 21ct century equivalent of a 1940's…

"Imagine being the 21ct century equivalent of a 1940's US Marine on Guadal Canal who need to know some survival skills. ChatGPT-on-a-phone would be handy if you could keep the battery charged."

Would it not make sense to learn survival skills...beforehand?

I would much rather learn survival skills in a controlled environment instead waiting until I need them and then whipping out the ChatGPT enabled phone for help...

Also, I don't need to have an entire library of knowledge with me, when a simple book of survival skills would be more than adequate...

I think pretty much every military in the world agrees with this.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#186

Post by bzmrgonz »

egonschiele wrote: post #81
yankoff wrote: post #43

Earlier quoted context omitted.

Is this opinion based on some benchmarking you (or someone else) did?

I believe there are benchmarks, but I can informally second that opinion. I'm building a writing app (chiseleditor.com) and there is nothing as good as the ChatGPT models right now.

Since you have your hands in the mess, let me ask you this, and I ask, because I think this is what is meant by people who ask what's a.. bla..bla alternative.. to bla..bla..bla. How can an industry specific or company specific AI be created? meaning you take the LLM engine and you ingress company data.. or if you want to be bold, industry datasets. CHATGPT is marketted as being doctor/architect/lawyer/professor/etc. But what if all you want to do is build an ask jeeve's type of ai lawyer??

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#187

Post by ukuina »

danpalmer wrote: post #101
segmondy wrote: post #22

Earlier quoted context omitted.

... strange. I'm running 30B models on a 10yr old PC with a $400 RTX 3060. Folks can run the 65B models with 4090 or dual 3090. Usually for about a cost of $2500.

GPT-3.5 is up to 175B parameters, GPT-4 (which is what OP is asking for) has been speculated as having 1T parameters, although that seems a little high to me. It's easy to run a much worse model on much worse hardware, but there's a reason why it's only companies with huge datacenter investments running the top models.

The 1T number is wild hype. It's likely GPT4 is actually a mixture-of-experts with smaller models under the hood.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#188

Post by MacsHeadroom »

VadimPR wrote: post #72
MacsHeadroom wrote: post #54

Earlier quoted context omitted.

> It's certainly not on a level with ChatGPT, but what is? Guanaco-65B is per https://arxiv.org/abs/2305.14314 CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF 4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQ

How much ram/vram required?

At least 48GB of RAM to run with llama.cpp, which now has CUDA GPU Offloading support.

The more you can fit on your GPU (in VRAM) the better (for speed), but no GPU is strictly required.

Post reply on HN