Live data from Hacker News

Show HN: I made a website to semantically search ArXiv papers

papermatch.mitanshu.tech

81–90 of 107 posts

Re: Show HN: I made a website to semantically search ArXiv papers

#81

Earlier quoted context omitted.

Why?

Not sure about the semantic search, but preprints are peer reviewed and hence not vetted. However, at the current pace of papers on arXiv (5k+/week) peer review alone might halt the progress.

You mean to say that preprints are not peer reviewed.

Re: Show HN: I made a website to semantically search ArXiv papers

#82
Instead of using binarized hamming, why not just use a shorter embedding that you can properly tackle? What good is Milvus if it's not giving you matches using something more proper?

Also, this site is not Reddit. You don't have to reply to every comment.

Re: Show HN: I made a website to semantically search ArXiv papers

#83
This is great! I just tried some queries and the results were pretty decent, in terms of semantics. But, just thinking of it as a user, if this were to be part of my daily workflow (instead of say something like Google Scholar), I would like:

1. The option to somehow see _how_ the paper was reviewed and/or cited, if at all. There are things like OpenReview, see example [1]

2. The ability to "tell me a story to get up to speed" about a collection of papers. Generative models could help here -- but essentially, I want this thing to be able to write a paragraph for what one might find in the literature review / related work of a paper, with citations. :-)

All the best!

[1] https://openreview.net/forum?id=jhKbnNhwhc

Re: Show HN: I made a website to semantically search ArXiv papers

#84
post #5

I enjoy seeing projects like this! If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, unfortunately the big publishers (Elsevier and Springer) are forcing other indices like OpenAlex, etc. to remove abstracts so they're harder to get. Have you checked out other tools like undermind.ai, scite.ai, and elicit.org? You might consider what else a dedicated product workflow for lit reviews inc…

There's always [redacted due to copyright infringement policy].se?

Re: Show HN: I made a website to semantically search ArXiv papers

#86
post #5

I enjoy seeing projects like this! If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, unfortunately the big publishers (Elsevier and Springer) are forcing other indices like OpenAlex, etc. to remove abstracts so they're harder to get. Have you checked out other tools like undermind.ai, scite.ai, and elicit.org? You might consider what else a dedicated product workflow for lit reviews inc…

Edit: I moved this here from top level.

The Cloudflare challenge screen at the beginning is a dealbreaker.

Random question - does anyone know why so many papers are missing from ArXiv? Do they need to be submitted manually, perhaps by their author(s)? I'll often find papers on mathematics, physics and computer science. But papers on biology, chemistry and medicine are usually missing.

I think a database of all paper ids in existence and where they're posted or missing could be at least as useful as this. Because no papers written with any level of public funding (meaning most of them) should ever be missing.

Re: Show HN: I made a website to semantically search ArXiv papers

#87

This is great! I just tried some queries and the results were pretty decent, in terms of semantics. But, just thinking of it as a user, if this were to be part of my daily workflow (instead of say something like Google Scholar), I would like: 1. The option to somehow see _how_ the paper was reviewed and/or cited, if at all. There are things like OpenReview, see example [1] 2. The ability to "tell me a story to get up…

1. I was not aware of OpenReview. I love the transparency and would definitely look into integrating it.

2. This is good feedback, making models write the Introduction section! I was planning to keep this search engine a little more traditional, however if the results are good, then it should be the way forward.

Thank you, Happy Holidays! :D

Re: Show HN: I made a website to semantically search ArXiv papers

#88

This is great! I just tried some queries and the results were pretty decent, in terms of semantics. But, just thinking of it as a user, if this were to be part of my daily workflow (instead of say something like Google Scholar), I would like: 1. The option to somehow see _how_ the paper was reviewed and/or cited, if at all. There are things like OpenReview, see example [1] 2. The ability to "tell me a story to get up…

1. I was not aware of OpenReview. I love the transparency and would definitely look into integrating it. 2. This is good feedback, making models write the Introduction section! I was planning to keep this search engine a little more traditional, however if the results are good, then it should be the way forward. Thank you, Happy Holidays! :D

I have to second the idea, having hacked together something similar myself, to help me complete a literature review——a literature review that I wasn’t planning to publish. Simply generating summaries or pulling key quotes, paper by paper, wasn’t sufficient to be able to understand the topic in the way I wanted to for writing the literature review. In the end, the system would process a collection of hundreds of PDFs that might be related, generate summaries of what they mentioned about the topic in question, and, importantly, was also prompted to note anything about how the insights built upon or were related to insights from previous research, and the motivations behind developing that insight / the challenge it was attempting to solve and whether it was successful. This worked well enough to reduce what might have been weeks worth of work to just a few hours. Genuinely, I believe that research in the near future could look a lot different from what it looks like today.

Re: Show HN: I made a website to semantically search ArXiv papers

#89
Feedback: first thing I tried is searching for "leaky relu" and I got a bunch of results related to fluids, which is... not very relevant. (:

Compare that to scholar which returns all relevant results:

https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q=leak...

You might want to retrain/finetune your own embedding model instead of using a general-purpose one.

Post reply on HN