Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I made a website to semantically search ArXiv papers

papermatch.mitanshu.tech

Post reply on HN
Print view View original

11–20 of 107 posts

Re: Show HN: I made a website to semantically search ArXiv papers

#11

Post by dmezzetti »

Excellent project.

As mentioned in another comment, I've put together an embeddings database using the arxiv dataset (https://huggingface.co/NeuML/txtai-arxiv) recently.

For those interested in the literature search space, a couple other projects I've worked on that may be of interest.

annotateai (https://github.com/neuml/annotateai) - Annotates papers with LLMs. Supports searching the arxiv database mentioned above.

paperai (https://github.com/neuml/paperai) - Semantic search and workflows for medical/scientific papers. Built on txtai (https://github.com/neuml/txtai)

paperetl (https://github.com/neuml/paperetl) - ETL processes for medical and scientific papers. Supports full PDF docs.

Re: Show HN: I made a website to semantically search ArXiv papers

#12

Post by shishy »

dmezzetti wrote: post #11

Excellent project. As mentioned in another comment, I've put together an embeddings database using the arxiv dataset ( https://huggingface.co/NeuML/txtai-arxiv ) recently. For those interested in the literature search space, a couple other projects I've worked on that may be of interest. annotateai ( https://github.com/neuml/annotateai ) - Annotates papers with LLMs. Supports searching the arxiv database mentioned ab…

paperetl is cool, saving that for later, nice! did something similar in-house with grobid in the past (great project by patrice).

Re: Show HN: I made a website to semantically search ArXiv papers

#13

Post by namanyayg »

What are other good areas where semantic search can be useful? I've been toying with the idea for a while to play around and make such a webapp.

Some of the current ideas I had:

1. Online ads search for marketers: embed and index video + image ads, allow natural language search to find marketing inspiration. 2. Multi e-commerce platform search for shopping: find products across Sephora, zara, h&m, etc.

I don't know if either are good enough business problems worth solving tho.

Re: Show HN: I made a website to semantically search ArXiv papers

#14

Post by dmezzetti »

shishy wrote: post #12
dmezzetti wrote: post #11

Excellent project. As mentioned in another comment, I've put together an embeddings database using the arxiv dataset ( https://huggingface.co/NeuML/txtai-arxiv ) recently. For those interested in the literature search space, a couple other projects I've worked on that may be of interest. annotateai ( https://github.com/neuml/annotateai ) - Annotates papers with LLMs. Supports searching the arxiv database mentioned ab…

paperetl is cool, saving that for later, nice! did something similar in-house with grobid in the past (great project by patrice).

Grobid is great. paperetl is the workhorse of the projects mentioned above. Good ole programming and multiprocessing to churn through data.

Re: Show HN: I made a website to semantically search ArXiv papers

#15

Post by bubaumba »

namanyayg wrote: post #13

What are other good areas where semantic search can be useful? I've been toying with the idea for a while to play around and make such a webapp. Some of the current ideas I had: 1. Online ads search for marketers: embed and index video + image ads, allow natural language search to find marketing inspiration. 2. Multi e-commerce platform search for shopping: find products across Sephora, zara, h&m, etc. I don't know i…

3. Quick lookup into internal documents. Almost any company needs it. Navigating file-system like hierarchy is slow and limited. That was old way.

4. Quick lookup into the code to find relevant parts even when the wording in comments is different.

Re: Show HN: I made a website to semantically search ArXiv papers

#16

Post by Quizzical4230 »

shigeru94 wrote: post #2

Is this similar to https://www.semanticscholar.org (from Allen Institute for AI) ?

It is more like what triilman commented, but with all components open-source. I plan to add filters soon enough with keywords support! (actually waiting for milvus)

Re: Show HN: I made a website to semantically search ArXiv papers

#17

Post by Quizzical4230 »

mrjay42 wrote: post #7

I think you have an encoding problem If you search for "UPC high performance computing evaluation", you'll see paper with buggy characters in the authors name (second results with that search).

Most definitely. Thank you for pointing this out!

Re: Show HN: I made a website to semantically search ArXiv papers

#19

Post by Quizzical4230 »

shishy wrote: post #5

I enjoy seeing projects like this! If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, unfortunately the big publishers (Elsevier and Springer) are forcing other indices like OpenAlex, etc. to remove abstracts so they're harder to get. Have you checked out other tools like undermind.ai, scite.ai, and elicit.org? You might consider what else a dedicated product workflow for lit reviews inc…

Thank you for the appreciation and great feedback!

| If you expand beyond arxiv, keep in mind since coverage matters for lit reviews,

I do have PaperMatchBio [^1] for bioRxiv and PaperMatchMed [^2] for medRxiv, however I do agree having multiple sites for domains isn't ideal. And I am yet to create a synchronization pipeline for these two so the results may be a little stale.

| unfortunately the big publishers (Elsevier and Springer) are forcing other indices like OpenAlex, etc. to remove abstracts so they're harder to get.

This sounds like a real issue in expanding the coverage.

| Have you checked out other tools like undermind.ai, scite.ai, and elicit.org?

I did, but maybe not thoroughly enough. I will check these and add complementing features.

| You might consider what else a dedicated product workflow for lit reviews includes besides search

Do you mean a reference management system like Mendeley/Zotero?

[1]: https://papermatchbio.mitanshu.tech/ [2]: https://papermatchmed.mitanshu.tech/

Post reply on HN