Nice. Why not use a full-text search like self-hosted Typesense?
Show HN: I made a website to semantically search ArXiv papers
31–40 of 107 posts
Re: Show HN: I made a website to semantically search ArXiv papers
#32Nice. Why not use a full-text search like self-hosted Typesense?
Full text search would be redundant as arXiv.org already supports it. For semantic search, Typesense has limited collection of embedding models. [^1]
Re: Show HN: I made a website to semantically search ArXiv papers
#33For every application of semantic search, I’d love to see what the benefit is over text search. If there a benchmark to see if it improves the search. Subjectively, did you find it surfaced new papers? Is this more useful in certain domains?
All benefits depend on the ability of the embedding model. Semantic embeddings understand nuances, so they can match abstracts that align conceptually even if no exact keywords overlap. For example, "neural networks" vs. "deep learning." can and should fetch similar papers.
Subjectively, yes. I sent this around my peers and they said it helped them find new authors/papers in the field while preparing their manuscripts.
| Is this more useful in certain domains?
I don't think I have the capacity to comment on this.
Re: Show HN: I made a website to semantically search ArXiv papers
#34Earlier quoted context omitted.
Thank you for the appreciation and great feedback! | If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, I do have PaperMatchBio [^1] for bioRxiv and PaperMatchMed [^2] for medRxiv, however I do agree having multiple sites for domains isn't ideal. And I am yet to create a synchronization pipeline for these two so the results may be a little stale. | unfortunately the big publishers (Elsev…
Unusual use case but I write literature reviews for French R&D tax cut system, and we specifically need to: focus on most recent papers, stay on topic for a very specific problematic a company has, potentially include grey literature (tech blog articles from renowned corp), be as exhaustive as possible when it comes to freely accessible papers (we are more ok with missing paid papers unless they are really popular).…
This is quite unique. I believe a custom solution might help you better than Google Scholar.
Re: Show HN: I made a website to semantically search ArXiv papers
#35Nice but I have to point out that a systematic review cannot be done with semantic search and should never be done in a preprint collection.
Why?
Not sure about the semantic search, but preprints are peer reviewed and hence not vetted. However, at the current pace of papers on arXiv (5k+/week) peer review alone might halt the progress.
Re: Show HN: I made a website to semantically search ArXiv papers
#36Re: Show HN: I made a website to semantically search ArXiv papers
#37This is awesome! If you’re interested, you could add a search tool client for your backend in paper-qa (https://github.com/Future-House/paper-qa). Then paper-qa users would be able to use your semantic search as part of its workflow.
Re: Show HN: I made a website to semantically search ArXiv papers
#38This is cool, but how about local semantic search through tens of thousands articles and books. Sure I'm not the first, there should be some tools already.
I definitely was thinking about something like this for PaperMatch itself. Where anyone can pull a docker image and search through the articles locally! Do you think this idea is worthwhile pursuing?
Absolutely worth doing. Here is interesting related video, local RAG:
https://www.youtube.com/watch?v=bq1Plo2RhYI
I'm not an expert, but I'll do it for learning. Then open source if it works. As far as I understand this approach requires a vector database and LLM which doesn't have to be big. Technically it can be implemented as local web server. Should be easy to use, just type and get a sorted by relevance list.
Re: Show HN: I made a website to semantically search ArXiv papers
#39This is awesome! If you’re interested, you could add a search tool client for your backend in paper-qa ( https://github.com/Future-House/paper-qa ). Then paper-qa users would be able to use your semantic search as part of its workflow.
paper-qa looks pretty cool. I will do so!
Re: Show HN: I made a website to semantically search ArXiv papers
#40Earlier quoted context omitted.
I definitely was thinking about something like this for PaperMatch itself. Where anyone can pull a docker image and search through the articles locally! Do you think this idea is worthwhile pursuing?
Absolutely worth doing. Here is interesting related video, local RAG: https://www.youtube.com/watch?v=bq1Plo2RhYI I'm not an expert, but I'll do it for learning. Then open source if it works. As far as I understand this approach requires a vector database and LLM which doesn't have to be big. Technically it can be implemented as local web server. Should be easy to use, just type and get a sorted by relevance list.
Perfect!
Although, atm I am only using retrieval without any LLM involved. Might try integrating if it significantly improves UX without compromising speeds.