Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I put PubMed in a vector DB

pubmedisearch.com

Post reply on HN
Print view View original

1–10 of 28 posts

Show HN: I put PubMed in a vector DB

#1

Post by mpmisko »

Hi HN,

As a researcher, I often found myself struggling with the limitations of keyword-based search when exploring PubMed papers. To address this, I created PubMed Search (https://www.pubmedisearch.com/), a tool that leverages a vector database to enable semantic search across medical research literature.

Some key features:

* Daily updates to ensure access to the latest articles

* Semantic search using latest & greatest embedding models

* Some additional useful info about the papers (tldr, journal, publication date, etc.)

Hope you find it useful!

Show HN: I put PubMed in a vector DB
pubmedisearch.com

Re: Show HN: I put PubMed in a vector DB

#2

Post by rkwz »

Congrats on shipping!

I'm curious how the search results rankings work, doesn't look like it's based on date or number of citations, but seems to be deterministic (persists over multiple searches). I did a keyword search using one word.

Re: Show HN: I put PubMed in a vector DB

#3

Post by mpmisko »

rkwz wrote: post #2

Congrats on shipping! I'm curious how the search results rankings work, doesn't look like it's based on date or number of citations, but seems to be deterministic (persists over multiple searches). I did a keyword search using one word.

Thanks!

It uses a vector search approach. Your query is embedded in a vector space using a language model and we find the closest vector to the query from the PubMed papers. This is a good summary of the techniques: https://learn.microsoft.com/en-us/azure/search/vector-search.... There are a couple more tricks but this is the gist.

The nice part is that this approach allows you to find relevant papers to your question. E.g, you can ask "Can secondhand smoke cause AMD?" and the very first few papers are answering your question (https://pubmedisearch.com/share/Can%20secondhand%20smoke%20c...). The more specific question, the better. :)

Re: Show HN: I put PubMed in a vector DB

#4

Post by victor106 »

mpmisko wrote: post #3
rkwz wrote: post #2

Congrats on shipping! I'm curious how the search results rankings work, doesn't look like it's based on date or number of citations, but seems to be deterministic (persists over multiple searches). I did a keyword search using one word.

Thanks! It uses a vector search approach. Your query is embedded in a vector space using a language model and we find the closest vector to the query from the PubMed papers. This is a good summary of the techniques: https://learn.microsoft.com/en-us/azure/search/vector-search... . There are a couple more tricks but this is the gist. The nice part is that this approach allows you to find relevant papers to your questi…

Cool idea!

What are some papers labeled "High Quality Article"? How do you determine that?

Re: Show HN: I put PubMed in a vector DB

#5

Post by mpmisko »

victor106 wrote: post #4
mpmisko wrote: post #3

Earlier quoted context omitted.

Thanks! It uses a vector search approach. Your query is embedded in a vector space using a language model and we find the closest vector to the query from the PubMed papers. This is a good summary of the techniques: https://learn.microsoft.com/en-us/azure/search/vector-search... . There are a couple more tricks but this is the gist. The nice part is that this approach allows you to find relevant papers to your questi…

Cool idea! What are some papers labeled "High Quality Article"? How do you determine that?

Just looking at stuff like citations and impact factors of journals.

Re: Show HN: I put PubMed in a vector DB

#8

Post by lucas_crocker »

This is very cool! 2 questions spring to mind:

1. How much did it cost to embed all those vectors and how many articles did you process? PMC is quite large.

2. Could elaborate a little more on your approach to ranking articles? Because I'm familiar with semantic search via embeddings put did you weight those with impact factors/citations? Like how does one even calculate that?

Anyhow, love the idea.

Re: Show HN: I put PubMed in a vector DB

#10

Post by mpmisko »

bdangubic wrote: post #9

Hey mate, should search by PMID work? Like 35982160 is PMID for "Rare coding variation provides insight into the genetic architecture and phenotypic context of autism" - not seeing this publication at all in search results...

Hi, it currently does not support search by PMID. But you can find the paper included in the results here (5th place):

https://pubmedisearch.com/share/Do%20some%20individuals%20wi...

Post reply on HN