Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I made a website to semantically search ArXiv papers

papermatch.mitanshu.tech

Post reply on HN
Print view View original

21–30 of 107 posts

Re: Show HN: I made a website to semantically search ArXiv papers

#21

Post by Quizzical4230 »

bubaumba wrote: post #9

This is cool, but how about local semantic search through tens of thousands articles and books. Sure I'm not the first, there should be some tools already.

I definitely was thinking about something like this for PaperMatch itself. Where anyone can pull a docker image and search through the articles locally! Do you think this idea is worthwhile pursuing?

Re: Show HN: I made a website to semantically search ArXiv papers

#23

Post by eric-burel »

Quizzical4230 wrote: post #19
shishy wrote: post #5

I enjoy seeing projects like this! If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, unfortunately the big publishers (Elsevier and Springer) are forcing other indices like OpenAlex, etc. to remove abstracts so they're harder to get. Have you checked out other tools like undermind.ai, scite.ai, and elicit.org? You might consider what else a dedicated product workflow for lit reviews inc…

Thank you for the appreciation and great feedback! | If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, I do have PaperMatchBio [^1] for bioRxiv and PaperMatchMed [^2] for medRxiv, however I do agree having multiple sites for domains isn't ideal. And I am yet to create a synchronization pipeline for these two so the results may be a little stale. | unfortunately the big publishers (Elsev…

Unusual use case but I write literature reviews for French R&D tax cut system, and we specifically need to: focus on most recent papers, stay on topic for a very specific problematic a company has, potentially include grey literature (tech blog articles from renowned corp), be as exhaustive as possible when it comes to freely accessible papers (we are more ok with missing paid papers unless they are really popular). A "dedicated product workflow" could be about taking business use cases like that into account. This is a real business problem, the Google Scholar lock up is annoying and I would pay for something better than what exists.

Re: Show HN: I made a website to semantically search ArXiv papers

#24

Post by Quizzical4230 »

antman wrote: post #8

Nice work. Any other technical comments, why did you use those embeddings, did you binarzue them, did you use any dpecial prompts?

At the beginning of the project, MixedBread's embedding model was small and leading the MTEB leaderboard [^1], hence I went with it.

Yes, I did binarize them for a faster search experience. However, I think the search quality degrades significantly after the first 10 results, which are same as fp32 search but with a shuffled order. I am planning to add a reranking strategy to boost better results upwards.

At the moment, this is plain search with no special prompts.

[1]: https://huggingface.co/spaces/mteb/leaderboard

Re: Show HN: I made a website to semantically search ArXiv papers

#26

Post by Quizzical4230 »

dmezzetti wrote: post #11

Excellent project. As mentioned in another comment, I've put together an embeddings database using the arxiv dataset ( https://huggingface.co/NeuML/txtai-arxiv ) recently. For those interested in the literature search space, a couple other projects I've worked on that may be of interest. annotateai ( https://github.com/neuml/annotateai ) - Annotates papers with LLMs. Supports searching the arxiv database mentioned ab…

Thank you for your kind words.

These look like great projects, I will surely check them out :D

Post reply on HN