Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Retrieval in LangChain

blog.langchain.dev

Post reply on HN
Print view View original

1–10 of 71 posts

Re: Retrieval in LangChain

#2

Post by 40291misteryo »

Thanks for this info! Regarding the first instance of a non-LangChain Retriever - the ChatGPT Retrieval Plugin. I know this was recently open sourced by OpenAI for their own purposes to help gain more traffic to them. Probably to help people figure out endpoints to hook into ChatGPT. But how about for other usages? Be interested in how this fits in with your vectorstores. Thanks for the read!

Re: Retrieval in LangChain

#4

Post by Spivak »

Really excited to see LangChain moving really fast in this space. They turn your favorite Llm into a real boy that can do real work.

Some "agents" in their vernacular that I've built.

* A reminder system that can take a completely free-form English description and turn it into a precise date and time and schedule it with an external scheduler.

* A tool that can take math either in English or ascii like y = 2x^2 + ln(x) and turn it into rendered LaTeX.

* A line editor that let's you ingest Word documents and suggests edits for specific sections.

* A chess engine.

Like it's crazy at just how trivial all this stuff is to build.

Re: Retrieval in LangChain

#5

Post by KrugerDunnings »

I've been playing around with sentence embeddings to search documents, but I wonder how useful they are as a natural language interface for a database. The way one might phrase a question might be very different content wise from how the document describes the answer. Maybe it might be possible to do some type of transform where the question is transformed into a possible answer and then turned into a embedding but I haven't found much info on that yet.

Another idea I've had is to "overfit" a generative model like GPT on a dataset but pay more attention to how url and the like are tokenised

Re: Retrieval in LangChain

#6

Post by nunodonato »

KrugerDunnings wrote: post #5

I've been playing around with sentence embeddings to search documents, but I wonder how useful they are as a natural language interface for a database. The way one might phrase a question might be very different content wise from how the document describes the answer. Maybe it might be possible to do some type of transform where the question is transformed into a possible answer and then turned into a embedding but I…

embeddings are really good at that, you dont need to use similar words at all.

Re: Retrieval in LangChain

#7

Post by IanCal »

KrugerDunnings wrote: post #5

I've been playing around with sentence embeddings to search documents, but I wonder how useful they are as a natural language interface for a database. The way one might phrase a question might be very different content wise from how the document describes the answer. Maybe it might be possible to do some type of transform where the question is transformed into a possible answer and then turned into a embedding but I…

> Maybe it might be possible to do some type of transform where the question is transformed into a possible answer and then turned into a embedding but I haven't found much info on that yet.

Here you go https://twitter.com/theseamouse/status/1614453236349693953

Re: Retrieval in LangChain

#8

Post by summarity »

KrugerDunnings wrote: post #5

I've been playing around with sentence embeddings to search documents, but I wonder how useful they are as a natural language interface for a database. The way one might phrase a question might be very different content wise from how the document describes the answer. Maybe it might be possible to do some type of transform where the question is transformed into a possible answer and then turned into a embedding but I…

> might phrase a question might be very different content wise from how the document describes the answer

That's what hypothetical embeddings solve: https://summarity.com/hyde

There are also encoding schemes for question-answer retrieval (e.g. ColBERT)

Re: Retrieval in LangChain

#9

Post by sharemywin »

KrugerDunnings wrote: post #5

I've been playing around with sentence embeddings to search documents, but I wonder how useful they are as a natural language interface for a database. The way one might phrase a question might be very different content wise from how the document describes the answer. Maybe it might be possible to do some type of transform where the question is transformed into a possible answer and then turned into a embedding but I…

is there an example what what your talking about?

Also would you just return a list of likely candidates and loop over the result set to see if any info is relevant to the question and then have the the final pass try to answer the question.

Post reply on HN