This looks great! I have used the biorXiv version of papermatch and it gives pretty good results!
Thank you for your kind words!
Show HN: I made a website to semantically search ArXiv papers
61–70 of 107 posts
Re: Show HN: I made a website to semantically search ArXiv papers
#621. why mixbread's model? 2. how much efficiency gain did you see binarising embeddings/using hamming distance? 3. why milvus over other vector stores? 4. did you automate the weekly metadata pull? just a simple cron job? anything else you need orchestrated? user thoughts on searching for "transformers on byte level not token level" - was good but didnt turn up https://arxiv.org/abs/2412.09871 also you might want more…
1. The model size was small enough to process the corpus fast-ish using the limited resources I have. They also support MRL and binary embeddings which help would be helpful in case I need to downsize on the VM size.
2. Close to 500ms. See [^1].
3. This [^2] was the reason I went with milvus. I also assumed that more stars would result in a bigger community and hence faster bug discovery and fixes. And better feature support.
4. Yes, I automated the weekly pull here [^3]. Since I am constrained on resources available, I used HuggingFace Spaces to do the automation for me :) Although, the space keeps sleeping and to avoid that, I am planning keep calling the same space using api/gradio_client. Let's see how that goes.
| which is more recent, more people might want
Absolutely agree. I am planning to add a 'Recency' sorting option for the same. It should balance between similarity and the date published.
| also you might want more result density - so perhaps a UI option to collapse the abstracts and display more in the first glance.
Oh, I will surely look into it. Thank you so much for a detailed response. :D
[1]: https://news.ycombinator.com/item?id=42507116#42509636 [2]: https://benchmark.vectorview.ai/vectordbs.html [3]: https://huggingface.co/spaces/bluuebunny/update_arxiv_embedd...
Re: Show HN: I made a website to semantically search ArXiv papers
#63Earlier quoted context omitted.
Thank you for the appreciation and great feedback! | If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, I do have PaperMatchBio [^1] for bioRxiv and PaperMatchMed [^2] for medRxiv, however I do agree having multiple sites for domains isn't ideal. And I am yet to create a synchronization pipeline for these two so the results may be a little stale. | unfortunately the big publishers (Elsev…
"|" it's a terrible character for signaling quotes, as it looks a bit too much like "I" or "l" and sometimes even "1" or "i" depending on the font used. I believe the greater-than symbol (>) is better suited for this task.
So true ;-;
I was following the Gmail protocol. I will use > from now on.
Happy Holidays :D
Re: Show HN: I made a website to semantically search ArXiv papers
#64For what it's worth, back in the day (a few years ago, before the LLM boom a few years) I found on a similar sized vector database (gensim / doc2vec), it's possible to just brute force a vector search e.g. with SSE or AVX type instructions. You can code it in C and have a python API. Your data appears to be a few gigs so that's feasible for realtime CPU brute force, <200 ms
This is an interesting problem to tackle. Added to TODO list! :D
Re: Show HN: I made a website to semantically search ArXiv papers
#65Very cool! Add a "similar papers" link to each paper, that will make this the obvious way to discover topics by clicking along the similar papers.
Amazing! I will do so :D
Re: Show HN: I made a website to semantically search ArXiv papers
#66Great procrastination project :)
hey hey hey! XD
Re: Show HN: I made a website to semantically search ArXiv papers
#67For every application of semantic search, I’d love to see what the benefit is over text search. If there a benchmark to see if it improves the search. Subjectively, did you find it surfaced new papers? Is this more useful in certain domains?
One of the factors is how users phrase their queries. On some level people are used to full text search but semantic shines when they ask literal questions with terminology that may not match the answer.
Exactly. Full text paradigm has it's own pros and I believe we need those tools in the new vector search to take full advantage. I am planning to add keywords feature where if a user enters something in "quotes", the would need to be in the shown results. Just like you can do with a google search.
Re: Show HN: I made a website to semantically search ArXiv papers
#68Earlier quoted context omitted.
That's has a major downgrade. For binary embeddings, the top 10 results are same as fp32, albeit shuffled. However after the 10th result, I think quality degrades quite a bit. I was planning to add a reranking strategy for binary embeddings. What do you think?
Recommend reranking. You basically get full resolution performance for a negligible latency hit. (Unless you need to make two network calls…) MixedBread supports matryoshka embeddings too so that’s another option to explore on the latency-recall curve.
> Recommend reranking.
Will explore it thoroughly then!
> MixedBread supports matryoshka embeddings too so that’s another option to explore on the latency-recall curve.
Yes, exactly why I went with this model!
Re: Show HN: I made a website to semantically search ArXiv papers
#69interesting project; I’m not really sure how useful it is for field-specific stuff—I'm searching for “image reduction astronomy”, and it shows all sorts of related but not image-reduction work (including noise reduction which is not the same thing). I’m not really familiar with vector search enough to evaluate it well enough. However I can give you the heads-up that the abstracts don't render well because (La)TeX is…
Yes, I think vector search is tricky to navigate at times since now the onus is on the user to explain the problem well. However, you can copy paste full abstracts to get similar papers well enough.
I will fix the LaTeX rendering ASAP.
Thank you for trying out the site! Happy Holidays :D
Re: Show HN: I made a website to semantically search ArXiv papers
#701. why mixbread's model? 2. how much efficiency gain did you see binarising embeddings/using hamming distance? 3. why milvus over other vector stores? 4. did you automate the weekly metadata pull? just a simple cron job? anything else you need orchestrated? user thoughts on searching for "transformers on byte level not token level" - was good but didnt turn up https://arxiv.org/abs/2412.09871 also you might want more…
1. The model size was small enough to process the corpus fast-ish using the limited resources I have. They also support MRL and binary embeddings which help would be helpful in case I need to downsize on the VM size. 2. Close to 500ms. See [^1]. 3. This [^2] was the reason I went with milvus. I also assumed that more stars would result in a bigger community and hence faster bug discovery and fixes. And better feature…
my pleasure, thank you for the reply! ive never used milvus or heard of mixbread so this was refreshing.