Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the best self hosted/local alternative to GPT-4?

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 206 posts

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#131

Post by laratied »

dustypotato wrote: post #62
_laiq wrote: post #32

Earlier quoted context omitted.

Incredible how people are underestimating ChatGPT or overestimating open-source models. A basic question: How can i join with SQL a column to a string separated with comma GPT-4: PostgreSQL: SELECT STRING_AGG(columnName, ', ') FROM tableName; Guanaco: Here is an example of how you could use the CONCAT function in MySQL to concatenate a string to a column value in a single-line query: SELECT CONCAT('The total price fo…

I have chatGPT with GPT-4 model. It never gives such concise answers. Is it because you used the API?

The default is rather wordy. You could instruct chatGPT4 to be less wordy or in the API put instructions to be less wordy in the system prompt or limit answers to a sentence.

I would assume the poster just edited this though.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#132

Post by inductive_magic »

bfeynman wrote: post #115
inductive_magic wrote: post #108

Earlier quoted context omitted.

OpenAI builds an LLM and an api-interface to that model. The design of abstractions, prompt engineering, custom fine-tunes and software engineering required to ship a valuable application on top of that interface counts as "building an app" in my book.

how can you finetune gpt-4 or 3.5? The shell/wrapper around a request to open AI is not any worthy piece of software, It'd be more impressive to vend those services. Otherwise it's basically just a few scripts.

As your wording implies, finetuning is restricted to the smaller models, i.e. babbage, curie etc.

You can generate the training data for this with 3.5 and 4 and tune smaller models with the resulting data. For lots of tasks, this results in robust results, which btw are also faster than 3.5 turbo.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#133

Post by cloverich »

rhacker wrote: post #122
simonh wrote: post #97

Earlier quoted context omitted.

Just to nitpick that a bit, they have a lead. To me, a moat is some other axis of advantage that makes it hard for a rival to compete directly. So for example Tesla has/had a lead in EV tech, but their supercharger network is a moat because it’s very difficult for a rival to compete with, even if they have equivalent EV tech. I know there isn’t established terminology for this. It is a nitpick, but I think a lead is…

So like, would that not suggest that Google has a moat with AI, as it is already intertwined with our email, calendars, etc... (if they ever make their AI public)

It makes it all the more interesting that Google is currently losing to an upstart. They have a MASSIVE advantage in data and moats, but cannot execute on novel products to take advantage of it, despite also hiring a significant chunk of the talent in the market as well.

But perhaps at this stage Google's not meant to / need to build cutting edge products, but rather focus on commoditizing the ones that prove profitable.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#134

Post by sqs »

wokwokwok wrote: post #30

There is literally no alternative. You’re stuck with openai, and you’re stuck with whatever rules, limitations or changes they give you. There are other models, but specifically if you’re actively using gpt-4 and find gpt-3.5 to be below the quality you require… Too bad. You’re out of luck. Wait for better open source models or wait patiently for someone to release a meaningful competitor, or wait for openai to relea…

Anthropic's Claude is on par with or better than GPT-4 for many tasks. It's not self-hosted, but it is competitive with OpenAI's best model.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#135

Post by edgyquant »

simonw wrote: post #69
seszett wrote: post #52

Earlier quoted context omitted.

My test for open models is surprisingly simple. I just ask "What is the capital of France?" and I haven't had a correct answer yet in any model I tried. They often get Paris right at least, but most other details are wrong. Guanaco says: > The current capital of France is Paris. It has been so since 1982 when it replaced the previous one which was Vichy.

I just got "The capital of France is Paris." from vicuna-v1-7B running entirely on my iPhone (using the MLC Chat app).

>They often get Paris right at least, but most other details are wrong.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#136

Post by inductive_magic »

icapybara wrote: post #120
inductive_magic wrote: post #108

Earlier quoted context omitted.

OpenAI builds an LLM and an api-interface to that model. The design of abstractions, prompt engineering, custom fine-tunes and software engineering required to ship a valuable application on top of that interface counts as "building an app" in my book.

The LLM _is_ the product. Everything else in the stack is window dressing. I don’t think OpenAI’s contribution should be so understated- they built a technology that was considered science fiction just a few years ago. They deserve all the credit for the “AI”.

I wasn't trying to understate OpenAI's contribution, far from it. I can hardly express my appreciation for their work.

That doesn't mean that everything else in the stack is window dressing though - custom, domain specific wrangling with the different api endpoints, finding a satisfying prompt, temperature param etc. for specific tasks - the entire process of designing systems around an LLM-api has many intricacies to it, lots of which are completely uncharted territory.

I can assure you: very smart people are knees deep in this process, and they deserve the credit for their share of the value that is being created.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#137

Post by forgingahead »

ngngngng wrote: post #125
icapybara wrote: post #120

Earlier quoted context omitted.

The LLM _is_ the product. Everything else in the stack is window dressing. I don’t think OpenAI’s contribution should be so understated- they built a technology that was considered science fiction just a few years ago. They deserve all the credit for the “AI”.

Good luck building a moat when 95% of your app is just calling the API everyone else has access to.

Paul English (founder of Kayak, exited at $2 billion) said this in response to the exact comment you are making: "Kayak was just a thin layer above ITA, and ITA sold for $700m and Kayak sold for $2b. So don’t dismiss thin layers. The ultimate power is the app and UX."[0]

I would also add "distribution" to the "app and UX" part, but you can certainly build a valuable business upon an API "everyone else has access to" - plenty of companies out there that do that.

[0]https://twitter.com/englishpaulm/status/1623701758781558784

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#138

Post by api »

_laiq wrote: post #32
jonathan-adly wrote: post #25

I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA). A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with rea…

Incredible how people are underestimating ChatGPT or overestimating open-source models. A basic question: How can i join with SQL a column to a string separated with comma GPT-4: PostgreSQL: SELECT STRING_AGG(columnName, ', ') FROM tableName; Guanaco: Here is an example of how you could use the CONCAT function in MySQL to concatenate a string to a column value in a single-line query: SELECT CONCAT('The total price fo…

Your question is a bit confusing and ambiguous. I think there are multiple valid answers. Here's a local test of a less ambiguous programming question with "Wizard-Vicuna-30B-Uncensored.ggmlv3.q8_0.bin" on llama.cpp on an M1 Max laptop with 64GiB of RAM.

This model seems roughly on par with GPT-3, maybe GPT-3.5 in some cases.

> Write a function to compute the maximum, minimum, and average of an array of doubles in C.

Here is an example function in C that computes the maximum, minimum, and average of an array of doubles:

```c void compute_statistics(double arr[], int n) { double max = arr[0]; double min = arr[0]; double sum = 0.0; for (int i = 1; i max) { max = arr[i]; } else if (arr[i] To use this function, simply call it with the array of doubles and its length as arguments. For example:

```c double arr[] = {1.0, 2.5, 3.0, 4.0, 5.0}; int n = sizeof(arr) / sizeof(arr[0]); compute_statistics(arr, n); ```

This will output:

``` Maximum: 5.00 Minimum: 1.00 Average: 3.00 ```

Here are some simple logic questions:

> Compute the sum of an array of 512 elements with each element having the value 2.

The sum of an array of 512 elements with each element having the value 2 is 1024.

> Compute the sum of an array of 512 elements with each element having the value 3.

The sum of an array of 512 elements with each element having the value 3 is 1536.

Not bad.

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#139

Post by angryasian »

ngngngng wrote: post #125
icapybara wrote: post #120

Earlier quoted context omitted.

The LLM _is_ the product. Everything else in the stack is window dressing. I don’t think OpenAI’s contribution should be so understated- they built a technology that was considered science fiction just a few years ago. They deserve all the credit for the “AI”.

Good luck building a moat when 95% of your app is just calling the API everyone else has access to.

more like utilizing prompts that anyone can figure out

Re: Ask HN: What's the best self hosted/local alternative to GPT-4?

#140

Post by forgingahead »

netsec_burn wrote: post #118
npteljes wrote: post #68

Earlier quoted context omitted.

I was wondering how much GPU memory does G-65B need, and from the docs, it's "48GB".

I'm running it on Serge with 51 GB of RAM. I don't think it requires any GPU memory (llama.cpp runs on the CPU), I have a RTX 2060 in the system I'm running it on. Correct me if I'm wrong.

How many tokens per second are you getting with your setup?
Post reply on HN