Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 134 posts

Re: Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data

#121

Post by yuhongsun »

baetylus wrote: post #115
ankit219 wrote: post #103

Earlier quoted context omitted.

I think search is the wrong lens to look at it. Yes, finding relevant information quickly is important, but the key to enterprise search tools would be to get a holistic view around any topic. A typical enterprise has a lot of silos (12 out of top 15 enterprise apps on G2 are addressing this problem) and the flow of information doesn't exist. Any enterprise search tool helps in aggregation and triangulation of conver…

>> the hard part is contextualizing it in a way it is immediately useful. That depends on understanding the company/domain lingo, understanding users, etc. How do you control this deterministically? It sounds like the "hard part" is variation in prompting & selectively choosing the right data to include, both of which I could see being good enough right now but hard to deliver definitively.

Being able to filter down the data deterministically is a big value add, especially as the number of documents scale into the range of multiple millions. We have filters by document-set, tags, time range, source type (ie. only include Slack + Google Drive, or Confluence + Jira + Gong, etc.)

The challenge is with the non-deterministic portions of the flow as you pointed out. Ensuring retrieval quality in out-of-domain datasets, guardrailing the LLM generation, working with conflicting or deprecated information are some of the interesting areas we're addressing. Happy to dive deeper on any aspect you're curious about, and I'm sure we can learn from the discussion as well.

Re: Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data

#122

Post by yuhongsun »

BillFranklin wrote: post #63
yuhongsun wrote: post #14

Earlier quoted context omitted.

I think I understand your concern but if I miss the point, please follow up! So regarding getting access to read knowledge from the different tools, it depends tool by tool but a lot of them have API keys or options for app integrations available in the free tier (GitHub, Google Drive, Confluence come to mind). Other tools don't have a free tier and you just get access to the API keys as a part of paying for the serv…

> It is what it is sadly. This is what I mean -- previously I built a similar search engine on top of slack, notion, etc., but didn't launch the product because I thought that requiring users to constantly add bots to private channels would be a subpar experience. I thought this would be a blocker for good UX, so didn't go further, but maybe you'll find a nice solution! Searching over public internal data is addresse…

Access controls is a non-glamorous but critical piece of what we're building. Currently implementing automatic access sync-ing for a few sources like Google Drive, Confluence, Jira, and Notion to start. By matching document-access in the source to users and groups, and then to emails, we can finally map Danswer users to document level access. So someone searching in Danswer will only get results based on the set of documents they have access to in the source tool.

For Slack it would look something like: get the users in the Slack channel, map those Slack users to users in Danswer. Then only those users in Danswer will be able to get results from that channel.

Re: Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data

#123

Post by yuhongsun »

dkarras wrote: post #109
yuhongsun wrote: post #73

Earlier quoted context omitted.

Hello, thanks for the kind words! With regards to your questions: 1. Are you referring to the local NLP models or the LLM? The local models are already open source models or ones we've trained ourselves. If you're talking about the LLM, the default is OpenAI but it's easy to configure other ones without any code changes. 2. Most sources are pulled from every 10 minutes. They have incremental updates so if you have Co…

how is it "chat over private data" if you are exposing my data to more parties like openai? I thought you were using a stack of self hosted open weight LLMs etc. If I can send it elsewhere, it is not private data.

So private refers to two things here, sorry for any confusion.

When we say "chat over private data" we mean that this data isn't publicly available and no LLMs have this knowledge in their training. Meaning that with our system you can now ask questions about team specific knowledge. For example, you can ask questions like "What features did customer X ask about in our last call". Obviously if you ask ChatGPT this, it will have no idea.

The other part is data privacy when using the system. The software can be plugged into most LLM providers or locally running LLMs. So if your team doesn't trust OpenAI but instead has a relationship with say Azure, or GCP, you can just plug into one of those instead. Alternatively, a lot of users recently have been setting up Danswer with locally running LLMs with tools like Ollama. In that case, you now have a truely airgapped system where no data is ever going outwards.

Re: Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data

#124

Post by yuhongsun »

beardedwizard wrote: post #98
yuhongsun wrote: post #40

Earlier quoted context omitted.

This is also another reason why we think OSS is the way to go here. Taking on the tech giants alone is definitely a daunting task (maybe even impossible for a small isolated team). The hope is that by working with the community, we'll be able to incorporate the best ideas and contributions from a large pool of like-minded people to build something everyone can benefit from! The OSS space has absolutely taken off in t…

What are the most exciting projects you have seen in this space so far?

I think the explosion in interest and all the new software around GenAI owe their success to NLP advancements (coming from an NLP background myself, I may be biased though). I think the best projects are those pushing the frontier of NLP and sharing the learnings with the world:

https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 - Mixture of Experts model like GPT-4 but this one is open source doesn't cost an arm and leg to run.

https://huggingface.co/mixedbread-ai/mxbai-rerank-xsmall-v1 - This is a very recent reranker, we will probably try it out. It's really nice that it's open source and small enough to run on CPU. I think an overall trend of the space in the next years will be to maintain/improve performance while shrinking the cost and hardware requirements.

Re: Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data

#125

Post by yuhongsun »

erlend_sh wrote: post #104
yuhongsun wrote: post #35

Earlier quoted context omitted.

Code search is something we have in our sights in the next couple months. Currently the GitHub connector pulls in PRs and Issues but not the whole code base. We wanted to have the best RAG pipeline so we deep dived on that. Code search uses a combination of graph based traversals and a different type of embedding so it's a separate effort, but we will definitely build it out since it's immensely useful to engineering…

I would hope you just partner with the open source Bloop AI for that :)

Ah thanks for pointing it out, it look like an awesome project! We love open source so it's nice to see others going this direction as well

Re: Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data

#126

Post by yuhongsun »

cyanydeez wrote: post #70
yuhongsun wrote: post #33

Earlier quoted context omitted.

A lot of people are very bullish on AI, it's very interesting to hear the opposite side. My opinion is that LLMs are very powerful at digesting and distilling knowledge which is why we built this project. I also think that LLMs are terrible reasoning engines and so agent-flows are not quite ready for primetime. Would love to hear your perspective on the space!

I certainly see the value of large document retrieval and various forms of search. However, what seems to be the business proposition is giving managers shallow access to documents but won't lead to rigorous information. There's a few middle grounds where it can yield insights. like regulatory scenarios where you want to understand how public orgs satisfy permits with written plans. however, what I don't believe will…

Ya, a lot of knowledge is locked away in metadata, how documents are structured, in non-textual representations, or sometimes even just in the head of experts. We definitely want to be more than a shallow access to documents and we're building with that in mind. We're currently working to include more metadata and organizational understanding, with plans to tackle OCR, NL-to-SQL, code search and knowledge graphs in the future. While we don't have concrete dates for all of these, hopefully this gives some visibility into our vision at least!

Regarding the context size, the research community is doing some stunning work. If you're interested, you should check out the new Mamba (Transformers replacement) architecture: https://arxiv.org/abs/2312.00752

Re: Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data

#127

Post by yuhongsun »

magaton wrote: post #119

Hello, congratulations. Danswer looks really interesting and the name is simply great. We are building something similar (internal enterprise search using llm) and I am thinking whether we should jump to Danswer codebase. I would have a couple of questions, if you could answer: - how would you compare Danswer and privateGPT? Do you see it as a direct competitor? - you posted below that you have not use llama hub conn…

Thanks for the kind words! Sorry for the delayed response, this post drew a lot more interest than anticipated and we've been swamped working with new folks coming in.

Regarding:

- PrivateGPT: they're for individual use where you ingest your own data. We're for teams to use with access controls, connectors to typical business SaaS tools, works at scale (incremental updates and scalable container architecture), different user roles, etc. So basically I see very little overlap between the two projects at least in terms of "competition", we're just different.

- LlamaHub: Our connectors pull all of the documents in the first run, then every following run, it only pulls in documents that have changed since the last run. For large teams, the first run may take many hours but following that, it will only take seconds each time. Without this, it becomes untenable to keep all information up to date. Also we pull in additional metadata and permissions, which not all LlamaHub connectors support.

- Push flow for indexing: Yes, we also have APIs that you can push to for indexing documents. For event based "push", we didn't go that route because most tools we connect to don't support this.

- Knowledge Graph: We will certainly be building this, it is only a matter of prioritization and timelines.

- Replacing the custom search: We think our search is much better than a basic RAG pipeline out of the box that someone can get from Langchain/LlamaIndex. What's the motivation for wanting to remove/replace it?

Re: Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data

#128

Post by yuhongsun »

cohnen wrote: post #117

Just wanted to drop in and share some love for Danswer. We've been using it as our go-to doc repository for a bit now, and it's been a game-changer for us. Not naming names, but let's just say it's powering some pretty key projects. What really blows my mind is the LLM capabilities. We're pulling out some seriously amazing answers from our knowledge base, making info retrieval a breeze. And yeah, I've been hands-on w…

Thanks for the kind words! There's nothing more motivating in the world than hearing that our users are loving what we've built!

Re: Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data

#129

Post by yuhongsun »

shetty wrote: post #116

Congrats on the launch! Finally got some time to try this out. Tried this on a couple of personal documents and compared to asking on chatGPT (both 4 and 3.5), so far the results weren't great. For context, the questions needed a little inference and danswer said the info isn't present in the document versus chatGPT which inferred the answer from a related statement. I do intend to perform a much bigger test around d…

Hi, it may be an indexing issue. There's a precanned "Information not found" message that we show if document retrieval failed. A couple common causes for this are:

- Not provisioning enough resources and processes are dying (we run NLP models locally so the system isn't totally lightweight) - Access is not correctly configured, either while pulling in from the source or at the user level during query time.

Assuming nothing is wrong with the setup, it is most likely because of how we prompt the LLM. Questions that require reasoning or are more open ended, generally aren't "safe" for the LLM to answer. So the system is constrained on purpose. For example if someone were to ask: "How can I increase revenue by 30% next quarter". It's not safe for the system to just propose some actions and it's likely better to just search the documents and say there wasn't any answer in the docs (unless of course some doc explicitly states plans for increasing revenue).

Re: Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data

#130

Post by yuhongsun »

sroussey wrote: post #79
Weves wrote: post #77

Earlier quoted context omitted.

Well the most standard approach is to use cross-encoders (e.g. something like Cohere Rerank) to give similarity scores between the query and the chunk, and then use these scores to update the ranking. Our approach is to use an LLM (gpt-3.5-turbo for example), and to ask it explicitly "Is this chunk useful for answering this query ". We've found, while certainly a bit more expensive, the larger model size and greater…

Interesting! Thanks for the explanation.

Two other tidbits on this:

1. There's a difference between relevance and usefulness that cross-encoders cannot capture. Imagine a thread with a bunch of people complaining about an exception and each comment is another mention of the exception. Now imagine another thread with one mention of the exception at the top, and a bunch of people offering solutions. If you query for the exception, LLMs will find the second thread more useful, but cross encoders will find the first one more relevant.

2. LLMs/GenAI models don't output a single value. They can use the tokens they output to "reason" about the usefulness of a doc. Eg. Rerankers are like tiny LLMs that are only allowed to output "yes" or "no", but instead you can use an LLM to do chain-of-thought and finally decide at the end.

Post reply on HN