Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run and create custom ChatGPT-like bots with OpenChat

github.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Run and create custom ChatGPT-like bots with OpenChat

#31

Post by iamflimflam1 »

forgingahead wrote: post #17
underlines wrote: post #16

Disclaimer: I am curating LLM-tools on github [1] A few thoughts: * allow for custom endpoint URLs, this way people can use open source LLMs with a fake openAI API backend like basaran[2] or llama-api-server[3] * look into better embedding methods for info-retrieval like InstructorEmbeddings or Document Summary Index * Don't use a single embedding per content item, use multiple to increase retrieval quality 1 https:/…

* Don't use a single embedding per content item, use multiple to increase retrieval quality Can you share some specific examples of what you mean by this? How would you process specific info types (eg: news article, or web page, or product catalogue data) this way, and how would you handle retrieval that makes the quality "better"? *Edit: Thanks for all replies so far - yes I am aware about splitting or chunking the…

There are a couple of things that can help. As has been pointed out by other commenters - chunking up content is very useful.

The other often neglected approach is to use an LLM to derive new content and use the embedding of this as well.

E.g. ask the LLM “give me a list of questions that can be answered by the following passage”

You then use embeddings of the generated questions instead of embeddings of the original content.

Re: Run and create custom ChatGPT-like bots with OpenChat

#32

Post by jerpint »

iamflimflam1 wrote: post #31
forgingahead wrote: post #17

Earlier quoted context omitted.

* Don't use a single embedding per content item, use multiple to increase retrieval quality Can you share some specific examples of what you mean by this? How would you process specific info types (eg: news article, or web page, or product catalogue data) this way, and how would you handle retrieval that makes the quality "better"? *Edit: Thanks for all replies so far - yes I am aware about splitting or chunking the…

There are a couple of things that can help. As has been pointed out by other commenters - chunking up content is very useful. The other often neglected approach is to use an LLM to derive new content and use the embedding of this as well. E.g. ask the LLM “give me a list of questions that can be answered by the following passage” You then use embeddings of the generated questions instead of embeddings of the original…

another approach is HyDE, where you ask the LLM to come up with a plausible (but likely wrong) answer, and use the embedding of the wrong answer to find the appropriate chunk, pretty clever

Re: Run and create custom ChatGPT-like bots with OpenChat

#34

Post by weinzierl »

jkukul wrote: post #5
ivanstegic wrote: post #3

This is a great idea and would love to see something like this succeed! If I understand how all of these OpenAI dependent apps work, none of them actually have the LLM and are doing any kind of heavy processing. AFAIK, they’re all packaging your data, submitting it to OpenAI on every request and then repackaging the output. There’s no real indexing, no real tangible thing, you have to start from scratch every time. S…

For most applications, packaging all the data and submitting it to OpenAI won't be feasible due to the limited token window size. I think the most common design pattern nowadays goes like this: 1. Chunk all your data (e.g. per paragraph of content) 2. Generate an embedding for each chunk 3. Index embeddings in a vector database 4. When a query comes in, find chunks relevant to the query (based on embeddings similarit…

I've seen this described as the common approach and argued for it but with my limited knowledge I have difficulties countering the argument that it would be best to just finetune the model with your own data.

I don't think it is as much the context window size because you would chunk your data anyways. I think the counter argument is either that finetuning is limited by the risk of overfitting and catastrophic forgetting or cost prohibitive. I think it is more of the former. Am I on the right track with this arguments?

Another point to consider is probably the vector DB contains an exact version of your data you get that as a result whereas the model will only be able answer vaguely or by paraphrasing.

Re: Run and create custom ChatGPT-like bots with OpenChat

#35

Post by weinzierl »

ivanstegic wrote: post #22
jkukul wrote: post #5

Earlier quoted context omitted.

For most applications, packaging all the data and submitting it to OpenAI won't be feasible due to the limited token window size. I think the most common design pattern nowadays goes like this: 1. Chunk all your data (e.g. per paragraph of content) 2. Generate an embedding for each chunk 3. Index embeddings in a vector database 4. When a query comes in, find chunks relevant to the query (based on embeddings similarit…

Thank you for this.

If you are interested in this approach, I found there are good examples using LangChain, so it's a good keyword to search for.

Re: Run and create custom ChatGPT-like bots with OpenChat

#36

Post by mrtranscendence »

lgas wrote: post #28
forgingahead wrote: post #17

Earlier quoted context omitted.

* Don't use a single embedding per content item, use multiple to increase retrieval quality Can you share some specific examples of what you mean by this? How would you process specific info types (eg: news article, or web page, or product catalogue data) this way, and how would you handle retrieval that makes the quality "better"? *Edit: Thanks for all replies so far - yes I am aware about splitting or chunking the…

I'm curious about more sophisticated answers to this question, but the obvious approach would be to split the article or web page into sentences and do an embedding per sentence.

When I was playing around with search via embeddings (as a test I was using Vampire the Masquerade V5 sourcebooks, and asking rules questions), I got the best results -- in terms of correct answers -- by using sentence embeddings. I'd search the query against the sentence embeddings, and then retrieve more context surrounding the winning sentence(s). That context would be passed to the LLM.

It wasn't perfect, though. I'm tempted to try the avenue of having an LLM generate questions for each passage and then use those embeddings, but it sounds a bit expensive to set up given the length of the books.

Re: Run and create custom ChatGPT-like bots with OpenChat

#37

Post by mrtranscendence »

experimenting wrote: post #19
jsherry wrote: post #13

This may be a naive question. But is there a way to embed a chat bot like this that only queries the data we feed it, and not the universe of other stuff in gpt? Like I don’t want people using the chatbot in our product to query a good strawberry shortcake recipe. We just want them to query about data we allow it to query which is native to our business. Is this feasible? Thx.

Yes, this is feasible. Look into https://github.com/NVIDIA/NeMo-Guardrails and specifically to your question there are "topical rails" to ensure the conversation stays on a set of topics you greenlighted. Also takes care of jailbreaks and allows custom conversation flow templates.

I'm curious how that works, as the documentation is a little under-specified. It seems like it requires specifying exact "utterances" from the user, but I don't think that can be the case -- wouldn't it be flatly useless that way? But it's not clear how to use it to, for example, disallow talking about politics. Or to disallow talking about topics unrelated to the dev's product, for that matter.

Re: Run and create custom ChatGPT-like bots with OpenChat

#39

Post by minimaxir »

jerpint wrote: post #33

> Each chatbot has unlimited memory capacity, enabling seamless interaction with large files such as a 400-page PDF. How is this possible? Do they do fancy tricks at inference?

Likely an embeddings vector store, which is not "memory" as traditionally defined in conversational AI apps.

Re: Run and create custom ChatGPT-like bots with OpenChat

#40

Post by lhenault »

underlines wrote: post #16

Disclaimer: I am curating LLM-tools on github [1] A few thoughts: * allow for custom endpoint URLs, this way people can use open source LLMs with a fake openAI API backend like basaran[2] or llama-api-server[3] * look into better embedding methods for info-retrieval like InstructorEmbeddings or Document Summary Index * Don't use a single embedding per content item, use multiple to increase retrieval quality 1 https:/…

Using this as an opportunity to mention my own related project, perhaps it can end up on your nice list one day. :)

https://github.com/lhenault/SimpleAI

Post reply on HN