Did you notice a difference in performance after binarization? Do you have a way to measure performance?
Show HN: I made a website to semantically search ArXiv papers
41–50 of 107 posts
Re: Show HN: I made a website to semantically search ArXiv papers
#42Did you notice a difference in performance after binarization? Do you have a way to measure performance?
Absolutely!
Here is a graph showing the difference. [^1]
Known ID is arXiv ID that is in the vector database, Unknown IDs need the metadata to be fetched via API. Text is embedded via the model's API.
FLAT and IVF_FLAT are different indexes used for the search. [^2]
[1]: https://raw.githubusercontent.com/mitanshu7/dumpyard/refs/he...
[2]: https://zilliz.com/learn/how-to-pick-a-vector-index-in-milvu...
Re: Show HN: I made a website to semantically search ArXiv papers
#43Did you notice a difference in performance after binarization? Do you have a way to measure performance?
Absolutely! Here is a graph showing the difference. [^1] Known ID is arXiv ID that is in the vector database, Unknown IDs need the metadata to be fetched via API. Text is embedded via the model's API. FLAT and IVF_FLAT are different indexes used for the search. [^2] [1]: https://raw.githubusercontent.com/mitanshu7/dumpyard/refs/he... [2]: https://zilliz.com/learn/how-to-pick-a-vector-index-in-milvu...
That looks great for speed, but what about recall?
Re: Show HN: I made a website to semantically search ArXiv papers
#44This looks great! I have used the biorXiv version of papermatch and it gives pretty good results!
Re: Show HN: I made a website to semantically search ArXiv papers
#45Earlier quoted context omitted.
Absolutely! Here is a graph showing the difference. [^1] Known ID is arXiv ID that is in the vector database, Unknown IDs need the metadata to be fetched via API. Text is embedded via the model's API. FLAT and IVF_FLAT are different indexes used for the search. [^2] [1]: https://raw.githubusercontent.com/mitanshu7/dumpyard/refs/he... [2]: https://zilliz.com/learn/how-to-pick-a-vector-index-in-milvu...
That looks great for speed, but what about recall?
That's has a major downgrade. For binary embeddings, the top 10 results are same as fp32, albeit shuffled. However after the 10th result, I think quality degrades quite a bit. I was planning to add a reranking strategy for binary embeddings. What do you think?
Re: Show HN: I made a website to semantically search ArXiv papers
#46What are other good areas where semantic search can be useful? I've been toying with the idea for a while to play around and make such a webapp. Some of the current ideas I had: 1. Online ads search for marketers: embed and index video + image ads, allow natural language search to find marketing inspiration. 2. Multi e-commerce platform search for shopping: find products across Sephora, zara, h&m, etc. I don't know i…
3. Quick lookup into internal documents. Almost any company needs it. Navigating file-system like hierarchy is slow and limited. That was old way. 4. Quick lookup into the code to find relevant parts even when the wording in comments is different.
For 4, it would be neat to first pass each block of code (function or class or whatever) through an llm to extract meaning, and then embed some combination of llm parsed meaning, docstring and comments, and function name. Then do semantic search against that.
That way you’d cover what the human thinks the block is for vs what an LLM “thinks” it’s for. Should cover some amount of drift in names and comments that any codebase sees.
Re: Show HN: I made a website to semantically search ArXiv papers
#47Earlier quoted context omitted.
That looks great for speed, but what about recall?
That's has a major downgrade. For binary embeddings, the top 10 results are same as fp32, albeit shuffled. However after the 10th result, I think quality degrades quite a bit. I was planning to add a reranking strategy for binary embeddings. What do you think?
Recommend reranking. You basically get full resolution performance for a negligible latency hit. (Unless you need to make two network calls…)
MixedBread supports matryoshka embeddings too so that’s another option to explore on the latency-recall curve.
Re: Show HN: I made a website to semantically search ArXiv papers
#48Very cool!
Add a "similar papers" link to each paper, that will make this the obvious way to discover topics by clicking along the similar papers.
Re: Show HN: I made a website to semantically search ArXiv papers
#49For every application of semantic search, I’d love to see what the benefit is over text search. If there a benchmark to see if it improves the search. Subjectively, did you find it surfaced new papers? Is this more useful in certain domains?
One of the factors is how users phrase their queries. On some level people are used to full text search but semantic shines when they ask literal questions with terminology that may not match the answer.
Re: Show HN: I made a website to semantically search ArXiv papers
#501. why mixbread's model?
2. how much efficiency gain did you see binarising embeddings/using hamming distance?
3. why milvus over other vector stores?
4. did you automate the weekly metadata pull? just a simple cron job? anything else you need orchestrated?
user thoughts on searching for "transformers on byte level not token level" - was good but didnt turn up https://arxiv.org/abs/2412.09871 also you might want more result density - so perhaps a UI option to collapse the abstracts and display more in the first glance.