Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I made a website to semantically search ArXiv papers

papermatch.mitanshu.tech

Post reply on HN
Print view View original

91–100 of 107 posts

Re: Show HN: I made a website to semantically search ArXiv papers

#91

Post by Quizzical4230 »

kouteiheika wrote: post #89

Feedback: first thing I tried is searching for "leaky relu" and I got a bunch of results related to fluids, which is... not very relevant. (: Compare that to scholar which returns all relevant results: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=leak... You might want to retrain/finetune your own embedding model instead of using a general-purpose one.

Thank you for taking the time to try out the site!

Google scholar scholar is a keyword based search engine. It looks for words as is in the text. PaperMatch tries to find similar papers that are closer in meaning.

Here is an alternative approach: Take one paper that you like, copy the abstract from Google Scholar and paste it in PaperMatch. This should help you find similar papers.

Re: Show HN: I made a website to semantically search ArXiv papers

#92

Post by Quizzical4230 »

zackmorris wrote: post #86
shishy wrote: post #5

I enjoy seeing projects like this! If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, unfortunately the big publishers (Elsevier and Springer) are forcing other indices like OpenAlex, etc. to remove abstracts so they're harder to get. Have you checked out other tools like undermind.ai, scite.ai, and elicit.org? You might consider what else a dedicated product workflow for lit reviews inc…

Edit: I moved this here from top level. The Cloudflare challenge screen at the beginning is a dealbreaker. Random question - does anyone know why so many papers are missing from ArXiv? Do they need to be submitted manually, perhaps by their author(s)? I'll often find papers on mathematics, physics and computer science. But papers on biology, chemistry and medicine are usually missing. I think a database of all paper…

> The Cloudflare challenge screen at the beginning is a dealbreaker.

I understand your concern, however, I do not have the know-how to properly combat bots that keep spamming the server and this seemed the easiest way for me to have a functional site. I would love to know some resources for beginners in this regard, if you have them.

>Random question...

arXiv is generally for submitting CS, maths and physics papers. There are alternate preprint repositories like biorxiv.org, chemrxiv.org and medrxiv.org for such purposes. Note: arxiv is the largest, in terms of papers hosted, among these.

Re: Show HN: I made a website to semantically search ArXiv papers

#94

Post by Quizzical4230 »

ProofHouse wrote: post #90

I couuld and really use this, but it didn't work for me. And HAS to have a date filter. That is a must maybe with some time based pre-option defaults like HackerNews. Good luck, want to try again when it works. Good idea

They are definitely planned to be integrated very soon! I probably should have waited to post on HN untill that. I will ping you once the features are live.

Thanks for trying out the site!

Re: Show HN: I made a website to semantically search ArXiv papers

#95

Post by Quizzical4230 »

zerop wrote: post #76

This looks great, thanks for building this. Something on similar lines which many may link, Research Rabbit - https://www.researchrabbit.ai/

I am glad you liked it!

I wanted PaperMatch to be open-source so that the users can understand the workflow behind it and hack it to their advantage instead of grumbling away when the results aren't to their liking.

Re: Show HN: I made a website to semantically search ArXiv papers

#96

Post by Quizzical4230 »

OutOfHere wrote: post #82

Instead of using binarized hamming, why not just use a shorter embedding that you can properly tackle? What good is Milvus if it's not giving you matches using something more proper? Also, this site is not Reddit. You don't have to reply to every comment.

> Also, this site is not Reddit. You don't have to reply to every comment.

I am so conflicted whether to reply to this comment or not Xp

Jokes apart, Mxbai model + Milvus gives fantastic results in fp32, however it's the latency that is an issue here. I could try chopping the fp32 vectors in half without binarizing to see. Thanks!

Re: Show HN: I made a website to semantically search ArXiv papers

#97

Post by zzyzek »

This seems like a cool idea, thanks for creating it!

Some feedback:

I tried searching for "wave function collapse algorithm", "gumin wave function collapse", "wfc" and "model synthesis" without any relevant hits to the area of research I was interested in. I got a lot of quantum computing and other physics related papers.

The "WFC algorithm" overloaded the term (and has nothing to do with quantum mechanics) so it's kind of a bad case for this type of search. Model synthesis is way too generic, so again, might be a bad case for this.

The first page of results using "wave function collapse algorithm" from arXiv itself gives relevant results.

Re: Show HN: I made a website to semantically search ArXiv papers

#98

Post by shishy »

zackmorris wrote: post #86
shishy wrote: post #5

I enjoy seeing projects like this! If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, unfortunately the big publishers (Elsevier and Springer) are forcing other indices like OpenAlex, etc. to remove abstracts so they're harder to get. Have you checked out other tools like undermind.ai, scite.ai, and elicit.org? You might consider what else a dedicated product workflow for lit reviews inc…

Edit: I moved this here from top level. The Cloudflare challenge screen at the beginning is a dealbreaker. Random question - does anyone know why so many papers are missing from ArXiv? Do they need to be submitted manually, perhaps by their author(s)? I'll often find papers on mathematics, physics and computer science. But papers on biology, chemistry and medicine are usually missing. I think a database of all paper…

There are other preprint servers. But to your question, there are centralized indices that track all papers.

DOI is the primary identifier and preprints are also issuing them now.

Crossref has papers by DOI. OpenAlex and SemanticScholar also have records, with different id types supported (doi, pmid, etc).

Re: Show HN: I made a website to semantically search ArXiv papers

#99

Post by Quizzical4230 »

zzyzek wrote: post #97

This seems like a cool idea, thanks for creating it! Some feedback: I tried searching for "wave function collapse algorithm", "gumin wave function collapse", "wfc" and "model synthesis" without any relevant hits to the area of research I was interested in. I got a lot of quantum computing and other physics related papers. The "WFC algorithm" overloaded the term (and has nothing to do with quantum mechanics) so it's k…

Thank you for taking the time to try out the site!

arXiv has a keyword based search engine. It looks for words as is in the text. PaperMatch tries to find similar papers that are closer in meaning.

Here is an alternative approach: Take one paper that you like, copy the abstract from arXiv (or arXiv ID) and paste it in PaperMatch. This should help you find similar papers.

Re: Show HN: I made a website to semantically search ArXiv papers

#100

Post by Quizzical4230 »

OutOfHere wrote: post #80
mskar wrote: post #37

This is awesome! If you’re interested, you could add a search tool client for your backend in paper-qa ( https://github.com/Future-House/paper-qa ). Then paper-qa users would be able to use your semantic search as part of its workflow.

I advise against it since binarized hamming distance isn't exactly that good unless your vector length is say a million.

I have the fp32 embeddings saved. It is for the website that I use binarised ones to combat latency.
Post reply on HN