Live data from Hacker News

Show HN: I made a website to semantically search ArXiv papers

papermatch.mitanshu.tech

51–60 of 107 posts

Re: Show HN: I made a website to semantically search ArXiv papers

#51

What are other good areas where semantic search can be useful? I've been toying with the idea for a while to play around and make such a webapp. Some of the current ideas I had: 1. Online ads search for marketers: embed and index video + image ads, allow natural language search to find marketing inspiration. 2. Multi e-commerce platform search for shopping: find products across Sephora, zara, h&m, etc. I don't know i…

Please stop making ad tech better. Someone else might, but you don’t have to.

Re: Show HN: I made a website to semantically search ArXiv papers

#52

Earlier quoted context omitted.

Unusual use case but I write literature reviews for French R&D tax cut system, and we specifically need to: focus on most recent papers, stay on topic for a very specific problematic a company has, potentially include grey literature (tech blog articles from renowned corp), be as exhaustive as possible when it comes to freely accessible papers (we are more ok with missing paid papers unless they are really popular).…

This is quite unique. I believe a custom solution might help you better than Google Scholar.

This can be seen as technology watch, as opposed to a thesis literature review for instance. Google Scholar gives the best results but sadly doesn't really want you to build products on top of it : no api, no scraping. Breaking this monopoly would be a huge step forward, especially when coupled with semantic search.

Re: Show HN: I made a website to semantically search ArXiv papers

#54
post #49

For every application of semantic search, I’d love to see what the benefit is over text search. If there a benchmark to see if it improves the search. Subjectively, did you find it surfaced new papers? Is this more useful in certain domains?

One of the factors is how users phrase their queries. On some level people are used to full text search but semantic shines when they ask literal questions with terminology that may not match the answer.

Query keyword expansion works quite well for that without semantic search (although it can reduce precision).

Re: Show HN: I made a website to semantically search ArXiv papers

#55
For what it's worth, back in the day (a few years ago, before the LLM boom a few years) I found on a similar sized vector database (gensim / doc2vec), it's possible to just brute force a vector search e.g. with SSE or AVX type instructions. You can code it in C and have a python API. Your data appears to be a few gigs so that's feasible for realtime CPU brute force, <200 ms

Re: Show HN: I made a website to semantically search ArXiv papers

#57
post #5

I enjoy seeing projects like this! If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, unfortunately the big publishers (Elsevier and Springer) are forcing other indices like OpenAlex, etc. to remove abstracts so they're harder to get. Have you checked out other tools like undermind.ai, scite.ai, and elicit.org? You might consider what else a dedicated product workflow for lit reviews inc…

Thank you for the appreciation and great feedback! | If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, I do have PaperMatchBio [^1] for bioRxiv and PaperMatchMed [^2] for medRxiv, however I do agree having multiple sites for domains isn't ideal. And I am yet to create a synchronization pipeline for these two so the results may be a little stale. | unfortunately the big publishers (Elsev…

"|" it's a terrible character for signaling quotes, as it looks a bit too much like "I" or "l" and sometimes even "1" or "i" depending on the font used. I believe the greater-than symbol (>) is better suited for this task.

Re: Show HN: I made a website to semantically search ArXiv papers

#58

Earlier quoted context omitted.

Thank you for the appreciation and great feedback! | If you expand beyond arxiv, keep in mind since coverage matters for lit reviews, I do have PaperMatchBio [^1] for bioRxiv and PaperMatchMed [^2] for medRxiv, however I do agree having multiple sites for domains isn't ideal. And I am yet to create a synchronization pipeline for these two so the results may be a little stale. | unfortunately the big publishers (Elsev…

Unusual use case but I write literature reviews for French R&D tax cut system, and we specifically need to: focus on most recent papers, stay on topic for a very specific problematic a company has, potentially include grey literature (tech blog articles from renowned corp), be as exhaustive as possible when it comes to freely accessible papers (we are more ok with missing paid papers unless they are really popular).…

Hey, I'm not OP, but I'm working on what seems to be the exact problem you mentioned. We (https://fixpoint.co/) search and monitor web data about companies. We are indexing patents and academic papers right now, plus we can scrape and monitor just about any website (some social media sites not supported).

We have users with very similar use cases to yours. Want to email me? dylan@fixpoint.co. I'm one of the founders :)

Re: Show HN: I made a website to semantically search ArXiv papers

#60
interesting project; I’m not really sure how useful it is for field-specific stuff—I'm searching for “image reduction astronomy”, and it shows all sorts of related but not image-reduction work (including noise reduction which is not the same thing). I’m not really familiar with vector search enough to evaluate it well enough.

However I can give you the heads-up that the abstracts don't render well because (La)TeX is interpreted as markdown so that

    Paper~1 shows something and Paper~2 shows something else
will strikethrough the text between the tildes (whereas they are meant to be non-breaking spaces). Similarly for the backtick which makes text monospaced in the rendered output but is simply supposed to be the opening quote.
Post reply on HN