Live data from Hacker News

Ask HN: Open-source ChatGPT alternatives?

news.ycombinator.com

31–40 of 64 posts

Re: Ask HN: Open-source ChatGPT alternatives?

#31

Earlier quoted context omitted.

You want a large language model. This gives you a large language model.

I asked for recommendations on which LLM would run on consumer hardware for the purposes of fine tuning and inference, with good results. You linked a package that can be used to download models? I don't see how these things are related.

Why don't go ask ChatGPT then?

But seriously I am asking a very similar question with a focus on LLMs for classification (e.g. "Is this an article about a sports game?"), information extraction, clustering and such. I am not so interested in generation (Which I am assuming you are.), however the GPT style embeddings and those are are useful for the kind of work I do and are interchangeable with BERT-like and other embeddings.

"Good" or "Best" is something you have to define for yourself and the one thing every successful A.I. developer has done is develop a facility for testing if the solution was performing acceptably. With that library you can download a model and start working with the model, again, the successful people all tested at least one model. In the time since your post, a run-of-the-mill Python developer could have made some progress. Learn Python or get a non technical co-founder,

For my kind of tasks I want something that handles bigger documents than ChatGPT and when I go shopping for models I cannot find a high quality very large transformer that has been assembled with a tokenizer and trained hard on language tasks. When I look at the literature it seems the very long transformers like reformer wouldn't perform so well if somebody did try to build an LLM so I wait. I am certain that somebody will upload a better model to huggingface someday -- that's the thought process it takes to get an answer for questions like yours.

If you look though at the process used to make ChatGPT-3 able to converse there is the GPT-style embedding and then a process of dialog generation trained on totally different principles which is the "Reinforcement Learning from Human Feedback"

https://www.assemblyai.com/blog/how-chatgpt-actually-works/

and I think you are not going to get that kind of capability open source in that the training data doesn't exist for it. There are many things you have to do once you have that training data, but I think there are many people able to follow that path now that it has been blazed.

Re: Ask HN: Open-source ChatGPT alternatives?

#32

Earlier quoted context omitted.

Maybe? GPT-J is closer to the AI-Dungeon model of intelligence. It's able to fill in the blank after what you type, but it's hysterically bad at answering precise questions (to the point that I had to nerf it for fun to see how stupid the output could get). It will handle basic natural language and context clues just fine. It's just not very fast, and the generations probably won't be as thorough as ChatGPT.

Am I going to be able to fine tune GPT-J or GPT-JT on consumer hardware?

When you say "fine tune" here what are you looking to do?

The impression I've got is that fine tuning large language models is mostly useful for very simple tasks, such as training a spam or categorization filter.

If you're looking to take a model and then e.g. train it on a few thousand additional pages of documentation in order to get it to answer project-specific questions, I've got the impression that fine tuning isn't actually a useful way to achieve that (I'd love to be proven wrong about this).

Instead, people are getting good results with "retrieval augmented generation" - where you first run a search (or an embeddings-based semantic search) against your docs to find relevant snippets, then feed them to the large language model as part of a glued together prompt.

I wrote about my explorations of this technique here - plenty of other people have written about this too: https://simonwillison.net/2023/Jan/13/semantic-search-answer...

Re: Ask HN: Open-source ChatGPT alternatives?

#33

Earlier quoted context omitted.

> Is this just naming and not relevant to the product at all? They took funding as an open source non profit. Once they got the money they turned into a closed sourced for-profit censorship machine.

Yeah, but to be fair GPT 2 is open source and Whisper (a high-quality speech recognition and multilingual translation model) is also open source. A few years ago I needed a good model for transcription for a project and I couldn’t find anything decent. They really have contributed to the open source community. If they keep releasing older models and keep their cutting-edge technology for profit I’m fine with it.

Fair enough.

Re: Ask HN: Open-source ChatGPT alternatives?

#34
GPT Neo 1.3B (https://huggingface.co/EleutherAI/gpt-neo-1.3B) is the largest I can run on my 12Gb VRAM GPU, and I'm sorry to say it's output is a joke (nowhere near GPT-3, more like GPT-2 level of BS).

However, you can fine tune it; and I'm sure with lots of fine tuning and some jiggling of the parameters you can get a half decent custom-purpose solution.

Re: Ask HN: Open-source ChatGPT alternatives?

#35
post #32

Earlier quoted context omitted.

Am I going to be able to fine tune GPT-J or GPT-JT on consumer hardware?

When you say "fine tune" here what are you looking to do? The impression I've got is that fine tuning large language models is mostly useful for very simple tasks, such as training a spam or categorization filter. If you're looking to take a model and then e.g. train it on a few thousand additional pages of documentation in order to get it to answer project-specific questions, I've got the impression that fine tuning…

> When you say "fine tune" here what are you looking to do?

As an example of fine tuning, I might take a pretrained model and then continue training it with a custom dataset that is tailored to a specific text generation task (not classification). Here is an example of a custom dataset that I might fine tune on:

https://github.com/baobabKoodaa/future/blob/8d2ae91e6a6f00c7...

I would like the LLM to generate fictional text in the same style as the fine tuned dataset.

Re: Ask HN: Open-source ChatGPT alternatives?

#36
post #26

In terms of models which are reasonably fast to run and easy to install, I think Flan-T5 is one of the best: https://huggingface.co/google/flan-t5-xxl - although out of the box it's more focused on giving short answers and it's very far from ChatGPT.

It's not clear from the link if it's possible to fine tune Flan-T5 on consumer hardware?

They released different sizes from "small" to "xxl", and at least "base" should be small enough to fine-tune virtually anywhere.

Re: Ask HN: Open-source ChatGPT alternatives?

#37

Open Assistant (started by some of the people that started Stable Diffusion I think?) is very early, but looks very promising. https://open-assistant.io/ https://github.com/LAION-AI/Open-Assistant

I briefly looked at this and it doesn't seem like they provide a model that I can fine tune on consumer hardware?

Hold your damn horses, this technology is brand new, requires a tremendous amount of data gathering and computation, there's a massive volunteer effort already under way, and you're begging for a free home version so you can save $0.02 cents. You want a hole, pick up a shovel.

Re: Ask HN: Open-source ChatGPT alternatives?

#38
post #18

Clue on conversation "history"-- "While ChatGPT is able to remember what the user has said earlier in the conversation, there is a limit to how much information it can retain. The model is able to reference up to approximately 3000 words (or 4000 tokens) from the current conversation - any information beyond that is not stored. Please note that ChatGPT is not able to access past conversations to inform its responses.…

Presumably one of the benefits of running your own model is that you can feed extra data into it via training rather than purely through inference? I.e. if you're a software company you could fine-tune it on your codebase, improving its answers without increasing inference time?

Re: Ask HN: Open-source ChatGPT alternatives?

#39
post #37

Earlier quoted context omitted.

I briefly looked at this and it doesn't seem like they provide a model that I can fine tune on consumer hardware?

Hold your damn horses, this technology is brand new, requires a tremendous amount of data gathering and computation, there's a massive volunteer effort already under way, and you're begging for a free home version so you can save $0.02 cents. You want a hole, pick up a shovel.

> Hold your damn horses, this technology is brand new, requires a tremendous amount of data gathering and computation, there's a massive volunteer effort already under way, and you're begging for a free home version so you can save $0.02 cents. You want a hole, pick up a shovel.

$0.02 cents per API request. What I would like to do is provide people a free service on the internet that using an LLM under the hood. If you are so rich that you can burn $10k on an internet hobby project, good for you, but just know that everybody else is not as rich as you.

Also I wasn't "begging for a free home version". In my original question I already provided one option. Sorry, I mean "one free home version". So it's not like I was starved out of options. There are options. I was asking for recommendations.

Re: Ask HN: Open-source ChatGPT alternatives?

#40
post #32

Earlier quoted context omitted.

When you say "fine tune" here what are you looking to do? The impression I've got is that fine tuning large language models is mostly useful for very simple tasks, such as training a spam or categorization filter. If you're looking to take a model and then e.g. train it on a few thousand additional pages of documentation in order to get it to answer project-specific questions, I've got the impression that fine tuning…

> When you say "fine tune" here what are you looking to do? As an example of fine tuning, I might take a pretrained model and then continue training it with a custom dataset that is tailored to a specific text generation task (not classification). Here is an example of a custom dataset that I might fine tune on: https://github.com/baobabKoodaa/future/blob/8d2ae91e6a6f00c7... I would like the LLM to generate fictional…

I've not yet managed to convince myself if fine tuning LLMs works for that kind of example.

Have you tried fine tuning GPT3 via the OpenAI APIs for this? It should only cost a few dollars for that smaller set of examples, and it would at least help demonstrate it it's possible to get the results you want with the current best-in-class language model before you try to run that against a smaller model that you can fit on your own hardware.

Post reply on HN