Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the best self hosted/local alternative to GPT-4?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 206 posts

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#101

Post by danpalmer »

segmondy wrote: post #22
danpalmer wrote: post #12

> Preferably self-hosted (I'm okay with paying for it) The big models, if even available, need >100GB of graphics memory to run and would likely take minutes to warm up. The pricing available via OpenAI/GCP/etc is only effective when you can multi-tenant many users. The cost to run one of these systems for private use would be ~$250k per year.

... strange. I'm running 30B models on a 10yr old PC with a $400 RTX 3060. Folks can run the 65B models with 4090 or dual 3090. Usually for about a cost of $2500.

GPT-3.5 is up to 175B parameters, GPT-4 (which is what OP is asking for) has been speculated as having 1T parameters, although that seems a little high to me.

It's easy to run a much worse model on much worse hardware, but there's a reason why it's only companies with huge datacenter investments running the top models.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#102

Post by born-jre »

There is so much parallel progress happening left and right at the same time they are not there yet. When things like sparseGPT and models fine-tuned with data with tool ability (not just instruct data) may be soon we get there, as long as there is progress i am hopeful. Some sort of inference optimized hardware would also help.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#103

Post by nabakin »

I would monitor and research each of these top models to determine which best fits your use case.

https://lmsys.org/blog/2023-05-25-leaderboard/

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

https://assets-global.website-files.com/61fd4eb76a8d78bc0676...

https://www.mosaicml.com/blog/mpt-7b

Also keep up to date with r/LocalLLaMA where new best open models are posted all the time.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#104

Post by philipov »

icapybara wrote: post #96
wokwokwok wrote: post #30

There is literally no alternative. You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you. There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require… Too bad. You’re out of luck. Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to relea…

This point is understated. So many people are going around like "I'm building an AI app!" when the reality is, OpenAI built an AI app, you're just designing a front end for it.

So, ChatGPT isn't a technology, it is a service.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#106

Post by evanchisholm »

wing-_-nuts wrote: post #85
MacsHeadroom wrote: post #65

Earlier quoted context omitted.

You link to Guanaco-33B, but Guanaco-65B is much more capable. CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF 4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQ

It irritates me to no end that people don't list the system requirements of various models. How much ram and vram does one need to run 4,13,33,65B models at a reasonable speed? edit: instead I'll ask this, what's the best model to run on a system with a 24gb 4090 and 64gb of ram?

It's fairly simple to estimate ram requirements based on parameter count: https://blog.eleuther.ai/transformer-math/

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#107

Post by weystrom »

https://github.com/oobabooga/text-generation-webui/

Works on all platforms, but runs much better on Linux.

Running this in Docker on my 2080Ti, can barely fit 13B-4bit models into 11G of VRAM, but it works fine, produces around 10-15 tokens/second most of the time. It also has an API, that you can use with something like LangChain.

Supports multiple ways to run the models, purely with CUDA (I think AMD support is coming too) or on CPU with llama.cpp (also possible to offload part of the model to GPU VRAM, but the performance is still nowhere near CUDA).

Don't expect open-source models to perform as well as ChatGPT though, they're still pretty limited in comparison. Good place to get the models is TheBloke's page - https://huggingface.co/TheBloke. Tom converts popular LLM builds into multiple formats that you can use with textgen and he's a pillar of local LLM community.

I'm still learning how to fine-tune/train LoRAs, it's pretty finicky, but promising, I'd like to be able to feed personal data into the model and have it reliably answer questions.

In my opinion, these developments are way more exciting than whatever OpenAI is doing. No way I'm pushing my chatlogs into some corp datacenter, but running locally and storing checkpoints safely would achieve my end-goal of having it "impersonate" myself on the web.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#108

Post by inductive_magic »

icapybara wrote: post #96
wokwokwok wrote: post #30

There is literally no alternative. You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you. There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require… Too bad. You’re out of luck. Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to relea…

This point is understated. So many people are going around like "I'm building an AI app!" when the reality is, OpenAI built an AI app, you're just designing a front end for it.

OpenAI builds an LLM and an api-interface to that model.

The design of abstractions, prompt engineering, custom fine-tunes and software engineering required to ship a valuable application on top of that interface counts as "building an app" in my book.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#109

Post by thomasfromcdnjs »

icapybara wrote: post #96
wokwokwok wrote: post #30

There is literally no alternative. You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you. There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require… Too bad. You’re out of luck. Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to relea…

This point is understated. So many people are going around like "I'm building an AI app!" when the reality is, OpenAI built an AI app, you're just designing a front end for it.

There is a lot of engineering work and reasoning to make sufficiently complex prompts.

Depending on your business case, the 4096 tokens given to you have to go quite far. Vector embeddings are not "easy" to work with. Trying to splat together a range of techniques to craft a good prompt is hard™.

Adding in Actions (e.g. using headless browsers to open pages etc) is also pioneering territory.

Sucks that OpenAI currently has the market, but there is still plenty of reasons to develop on top of it.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#110

Post by amilios »

npteljes wrote: post #68
amilios wrote: post #3

How much GPU memory do you have access to? If you can run it, Guanaco-65B is probably as close as you can get in terms of something publicly available. https://github.com/artidoro/qlora . But as other comments mention, it's still noticeably worse in my experience.

I was wondering how much GPU memory does G-65B need, and from the docs, it's "48GB".

Is this with the 4-bit quantization? The only issue with it is that inference is incredibly slow with it on right now, but that should be fixed up in the next few weeks I think.
Post reply on HN