Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Chat with your data using LangChain, Pinecone, and Airbyte

airbyte.com

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Show HN: Chat with your data using LangChain, Pinecone, and Airbyte

#51

Post by samspenc »

mtricot wrote: post #33
ramesh31 wrote: post #16

I don't know what datasets you guys are working with that have no issues being shared in plain text across three separate proprietary paid services, but this is a nonstarter for me.

When reading the tutorial, we are describing one stack to build a specific app. But the stack is made of building blocks that you can replace with others if you need to. - Airbyte has two self-hosted options: OSS & Enterprise - Langchain: OSS - OpenAI: you can host an OSS model if you want to - Pinecone: there are OSS/self-hosted alternatives

> - OpenAI: you can host an OSS model if you want to

Just to confirm: you mean models like Facebook's Llama 2 and variants right? Since OpenAI hasn't released any OSS models.

Re: Show HN: Chat with your data using LangChain, Pinecone, and Airbyte

#52

Post by fakedang »

rahimnathwani wrote: post #40
_pdp_ wrote: post #17

A fantastic starting point for beginners! Personally, I believe this tutorial provides a solid foundation, but there's so much more to explore. Building something truly effective involves tackling various nuanced situations and special cases. While querying records in Pinecone can sometimes give you the right results, it can also be a bit unpredictable, depending on what and how you query. You might want to check out…

I'm curious: did you have ChatGPT lightly edit this comment before posting? A few things about the style (like the final sentence) sound similar to GPT-4 output.

We are reaching peak HN. Animated discussions on everything by chatbots, while the humans are the lurkers.

Re: Show HN: Chat with your data using LangChain, Pinecone, and Airbyte

#53

Post by sandGorgon »

hi folks, when will you have pgvector as a destination ? we (https://github.com/arakoodev/edgechains) work with a lot of enterprises and they would not move away from using redis or pgvector even as their vector store. Is there a way where we can leverage that ?

Second, for a LOT of enterprises, they want to use non-openai embedding models (minilm, GTE, BGE), will you support that. For e.g. in Edgechains we natively support BGE and minilm. Would you be able to support that ?

Re: Show HN: Chat with your data using LangChain, Pinecone, and Airbyte

#54

Post by tomr75 »

johndhi wrote: post #36
mtricot wrote: post #8

Earlier quoted context omitted.

Shouldn't have any limits here. Can you let us know how it goes?

hmm, as a person of low technical savvy, do you expect there will be a point at which I can upload a large text file and have you do all the work to let me chat with it? I'd pay for that today if it exists, but can't put a ton of effort into building/implementing something myself.

chatpdf..?

Re: Show HN: Chat with your data using LangChain, Pinecone, and Airbyte

#55

Post by anupsurendran »

I feel that there are too many moving pieces here especially for prototyping. There was a much more simpler app recently I took a look at on a recent hackernews post : https://news.ycombinator.com/item?id=36894142

They still have work to do with different connectors (e.g. PDF etc) but the realtime simple document pipeline is what helps a lot.

Re: Show HN: Chat with your data using LangChain, Pinecone, and Airbyte

#56

Post by mtricot »

samspenc wrote: post #51
mtricot wrote: post #33

Earlier quoted context omitted.

When reading the tutorial, we are describing one stack to build a specific app. But the stack is made of building blocks that you can replace with others if you need to. - Airbyte has two self-hosted options: OSS & Enterprise - Langchain: OSS - OpenAI: you can host an OSS model if you want to - Pinecone: there are OSS/self-hosted alternatives

> - OpenAI: you can host an OSS model if you want to Just to confirm: you mean models like Facebook's Llama 2 and variants right? Since OpenAI hasn't released any OSS models.

correct

Re: Show HN: Chat with your data using LangChain, Pinecone, and Airbyte

#57

Post by zarazas »

jmorgan wrote: post #4

LangChain supports local LLMs like Llama 2 with Ollama ( https://github.com/jmorganca/ollama ) as of this morning, in both their Python and Javascript versions: https://python.langchain.com/docs/integrations/llms/ollama This can be a great option if you'd like to keep your data local versus submitting it to a cloud LLM, with the added benefit of saving costs if you're submitting many questions in a row (e.g. in batch…

Can llama 2 also be used to create the embeddings?

Re: Show HN: Chat with your data using LangChain, Pinecone, and Airbyte

#58

Post by zarazas »

mtricot wrote: post #7
swyx wrote: post #5

congrats team! what was the thinking behind choosing to support "Vector Database (powered by LangChain)" instead of directly supporting Pinecone Chroma et al directly as you do in the other destinations? when is direct integration the right approach vs when is it better to have an (possibly brittle, but faster time to market) integration of an integration?

Great question :) We want to get to value as fast as possible. I am certain that at some point we will need to go deeper with those integrations and they will likely require to be separate destinations. It will also depend on how they differentiate from each others, we will need more granularity with configurations.

I ak playing around with langchain the last days as well and when I checked right all langchain is really doing for you is giving you a guideline about recommended steps for a vector assisted LLM. In your example it actually just adds some text to the prompt like: "Answer the following question with the context provided here, If you dont find the right info dont make something up" sth. Along those lines

Re: Show HN: Chat with your data using LangChain, Pinecone, and Airbyte

#59

Post by zarazas »

mtricot wrote: post #33
ramesh31 wrote: post #16

I don't know what datasets you guys are working with that have no issues being shared in plain text across three separate proprietary paid services, but this is a nonstarter for me.

When reading the tutorial, we are describing one stack to build a specific app. But the stack is made of building blocks that you can replace with others if you need to. - Airbyte has two self-hosted options: OSS & Enterprise - Langchain: OSS - OpenAI: you can host an OSS model if you want to - Pinecone: there are OSS/self-hosted alternatives

What about the embedding?

Re: Show HN: Chat with your data using LangChain, Pinecone, and Airbyte

#60

Post by johndhi »

tomr75 wrote: post #54
johndhi wrote: post #36

Earlier quoted context omitted.

hmm, as a person of low technical savvy, do you expect there will be a point at which I can upload a large text file and have you do all the work to let me chat with it? I'd pay for that today if it exists, but can't put a ton of effort into building/implementing something myself.

chatpdf..?

chatpdf doesn't support my volume -- files are too big.
Post reply on HN