Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Your website's content -> Q&A bot / chatbot

github.com

Post reply on HN
Print view View original

21–28 of 28 posts

Re: Your website's content -> Q&A bot / chatbot

#21

Post by mpaepper »

kacperlukawski wrote: post #19
mpaepper wrote: post #14

Earlier quoted context omitted.

Yep, that's the way it's currently implemented in langchain. The 4 is a hyperparameter you can change, though, so you could set it to 10 as well. The way it works is that it first looks up the N most relevant documents (N being 4 in the default case) in the FAISS store relevant to the question, so it uses distance of embedding vectors for this lookup. Then it uses GPT3 to get summaries of the 4 entries related to the…

Looks interesting! Have you considered a proper vector database like Qdrant ( https://qdrant.tech )? FAISS runs on a single machine, but if you want to scale things up, then a real database makes it a lot easier. And with a free 1GB cluster on Qdrant Cloud ( https://cloud.qdrant.io ), you can store quite a lot of vectors. Qdrant is also already integrated with Langchain.

Thanks for the suggestion, but for my fun small experiment, FAISS was more than enough.

Re: Your website's content -> Q&A bot / chatbot

#23

Post by mpaepper »

rcarmo wrote: post #22

Curious to see if it can take my entire site content: https://taoofmac.com/static/graph Might be a fun weekend experiment.

Woah, that's a huge site!

Should be fine, though, as it iterates over it, it creates embeddings and then stores them in the FAISS store (https://github.com/facebookresearch/faiss) which was created to handle a large amount of embeddings.

For the actual queries, it filters it down by the most relevant documents which are closest in the embedding space, so this should work.

Let me know how it goes!

Re: Your website's content -> Q&A bot / chatbot

#25

Post by kordlessagain »

Guillaume86 wrote: post #20
kacperlukawski wrote: post #19

Earlier quoted context omitted.

Looks interesting! Have you considered a proper vector database like Qdrant ( https://qdrant.tech )? FAISS runs on a single machine, but if you want to scale things up, then a real database makes it a lot easier. And with a free 1GB cluster on Qdrant Cloud ( https://cloud.qdrant.io ), you can store quite a lot of vectors. Qdrant is also already integrated with Langchain.

Probably not very helpful at the scale most people would run this. Even brute forcing the search on CPU gives results in a few ms on small datasets.

Using something like Weaviate, which can be started in Docker with a one-liner, will give the ability to move away or toward dense vectors by concept. While doing dot product with manual code is fairly easy, using Weaviate to do the lifting (for embeddings as well) makes things super simple.

https://github.com/FeatureBaseDB/slothbot/blob/slothbot-work...

Re: Your website's content -> Q&A bot / chatbot

#26

Post by danielbln »

mpaepper wrote: post #15
layoric wrote: post #2

Nice to have tools like this to wrap up features, definitely makes these types of solutions more accessible, thanks! It would be nice to know from your experience if there is a kind of rule of thumb for calculating cost of fine tuning and running a solution like this against a docs site?

I don't have larger scale experience on this at the moment, but I can tell you what I observed during my trials (also see my related blog entry for some more info: https://www.paepper.com/blog/posts/build-q-and-a-bot-of-your... ) It cost around 0.05$ to create the embeddings for my ~50 blog entries. Asking a question in the way I've described it also costs around 0.05$ via the API.

0.05$ for a question seems expensive, are you using davinci3 or gpt3.5-turbo?

Re: Your website's content -> Q&A bot / chatbot

#27

Post by grogenaut »

kordlessagain wrote: post #25
Guillaume86 wrote: post #20

Earlier quoted context omitted.

Probably not very helpful at the scale most people would run this. Even brute forcing the search on CPU gives results in a few ms on small datasets.

Using something like Weaviate, which can be started in Docker with a one-liner, will give the ability to move away or toward dense vectors by concept. While doing dot product with manual code is fairly easy, using Weaviate to do the lifting (for embeddings as well) makes things super simple. https://github.com/FeatureBaseDB/slothbot/blob/slothbot-work...

that means you need docker running and the dependencies explode if you take this approach. I really like the tight dependency tree.

Re: Your website's content -> Q&A bot / chatbot

#28

Post by limcheekin »

Thanks for sharing the code. What happen when the existing content get updated and new contents created, would it need to create embeddings for all contents again? The current approach is not good as create embeddings cost money? Please see https://github.com/mpaepper/content-chatbot/blob/main/create.... Would it be possible progressively update the vector store?

Please advise. Thank you.

Post reply on HN