Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the best self hosted/local alternative to GPT-4?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 206 posts

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#31

Post by londons_explore »

jonathan-adly wrote: post #25

I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with rea…

Table 1 in your link (https://arxiv.org/pdf/2305.14314.pdf) is a good way to compare models.

I'd really like someone to make a big leaderboard/ranking engine which pits all these engines against eachother and publishes the resulting Elo score.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#32

Post by _laiq »

jonathan-adly wrote: post #25

I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with rea…

Incredible how people are underestimating ChatGPT or overestimating open-source models.

A basic question: How can i join with SQL a column to a string separated with comma

GPT-4: PostgreSQL: SELECT STRING_AGG(columnName, ', ') FROM tableName;

Guanaco: Here is an example of how you could use the CONCAT function in MySQL to concatenate a string to a column value in a single-line query: SELECT CONCAT('The total price for ', product_name, ’ is ', SUM(price)) AS total FROM products; This will result in output like “The total price for Chocolate Bar is 10” or similar depending on your data.

Yeah, no...

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#33

Post by jbellis »

segmondy wrote: post #22
danpalmer wrote: post #12

> Preferably self-hosted (I'm okay with paying for it) The big models, if even available, need >100GB of graphics memory to run and would likely take minutes to warm up. The pricing available via OpenAI/GCP/etc is only effective when you can multi-tenant many users. The cost to run one of these systems for private use would be ~$250k per year.

... strange. I'm running 30B models on a 10yr old PC with a $400 RTX 3060. Folks can run the 65B models with 4090 or dual 3090. Usually for about a cost of $2500.

Don't 4090 and 3090 have the same amount of vram?

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#34

Post by londons_explore »

catwalk_moto wrote: post #14

[dead]

Worth noting that Llama and derivatives most likely can be used for commercial use, despite what the licensing terms say. There seems to be growing consensus that model weights are not copyrightable, and therefore adhering to the terms of the license isn't required if you downloaded them from a third party.

Obviously being legally right doesn't necessarily save you from having Meta's legal team breathing down your neck...

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#35

Post by sounds »

Solvency wrote: post #29
AndroTux wrote: post #19

https://gpt4all.io/ works fairly well on my 16 GB M1 Pro MacBook. It's certainly not on a level with ChatGPT, but what is? It's a simple app download and allows you to select from multiple available models. No hacking required.

Why hasn't the community created a distributed Folding@home style GPT4/LLM software, where anyone can ask a question, and all participants machine's are contributing to the computation of the answer? It seems like the ideal way for the open source community. Is it not possible for some reason with the way LLMs work and compute? Or has it simply just not been done yet?

The model can run across multiple machines but it's so slow that way that nobody bothered to ship that.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#38

Post by airgapstopgap »

There really do not exist any alternatives, self-hosted or not. But more importantly, there may never be, what with the rising tide of AI risks and regulations discourse. It seems that soon training and opensourcing or otherwise making accessible a model of that class will be impossible, even as the cost of its production falls.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#39

Post by bigmattystyles »

wokwokwok wrote: post #30

There is literally no alternative. You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you. There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require… Too bad. You’re out of luck. Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to relea…

The model is the spice then basically?

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#40

Post by simonw »

Solvency wrote: post #29
AndroTux wrote: post #19

https://gpt4all.io/ works fairly well on my 16 GB M1 Pro MacBook. It's certainly not on a level with ChatGPT, but what is? It's a simple app download and allows you to select from multiple available models. No hacking required.

Why hasn't the community created a distributed Folding@home style GPT4/LLM software, where anyone can ask a question, and all participants machine's are contributing to the computation of the answer? It seems like the ideal way for the open source community. Is it not possible for some reason with the way LLMs work and compute? Or has it simply just not been done yet?

That exists: https://github.com/bigscience-workshop/petals
Post reply on HN