Launch HN: Danswer (YC W24) – Open-source AI search and chat over private data

Are you referring to the approach of creating hypothetical questions for embedding along with the document. Or more along the lines of creating summaries of the documents during indexing and embedding those as well?

Either case, the reason we don't do LLM based transformations during indexing time is that we hope our tool can be used by teams with a lot of documents. It becomes prohibitively expensive to run these transformations at indexing time when the scale can be multiple millions of documents.

Ya, I haven't dug too deep into their project, to be honest. But we do think it's great more teams are going for open source.

I tried to look up their RAG pipeline to see if they've invested effort into building a strong retrieval. It looks like they have a more basic vector only search, not sure if it's running local models or using a cloud service. On the flip side, it looks like they have assistants that can take actions (which we don't do), so maybe that's their focus.

I can only speculate as I've never spoken with their team but I imagine they're working closely with a few specific teams judging based on their number of connectors. Most likely it was built for specific customers. We are leaning more into the open source and working with our community and letting people self-host easily to find value with no friction.