What storage did you go for, and what search approach?
Show HN: I put PubMed in a vector DB
11–20 of 28 posts
Re: Show HN: I put PubMed in a vector DB
#12Nice! Out of curiosity what model(s) are you using to generate the embeddings?
Glad you like it! I did this as a mini-project within our startup MediSearch (https://medisearch.io/) & the search pipeline is custom tuned for the problem.
Re: Show HN: I put PubMed in a vector DB
#13What are you embedding exactly? Chunks of documents?
Yes
Re: Show HN: I put PubMed in a vector DB
#14This is very cool! 2 questions spring to mind: 1. How much did it cost to embed all those vectors and how many articles did you process? PMC is quite large. 2. Could elaborate a little more on your approach to ranking articles? Because I'm familiar with semantic search via embeddings put did you weight those with impact factors/citations? Like how does one even calculate that? Anyhow, love the idea.
1. We cover all the articles on PMC. The exact cost is hard to estimate because we did a lot of iterations.
2. We do weight those ... it is a lot of trial and error and you have to have good & exhaustive benchmarks.
Re: Show HN: I put PubMed in a vector DB
#15What storage did you go for, and what search approach?
We use pinecone and it is not ideal, looking at https://turbopuffer.com/ now. They look quite promising :)
Re: Show HN: I put PubMed in a vector DB
#16Very cool!
Related: the NIST TREC (Text REtrieval Conference) has had several competitions over the years related to improving the searchability of medical data: https://www.trec-cds.org/
If you have novel ideas in this area, you should consider participating. https://trec.nist.gov/
Re: Show HN: I put PubMed in a vector DB
#17Re: Show HN: I put PubMed in a vector DB
#18Very cool! Related: the NIST TREC (Text REtrieval Conference) has had several competitions over the years related to improving the searchability of medical data: https://www.trec-cds.org/ If you have novel ideas in this area, you should consider participating. https://trec.nist.gov/
Thanks! Looks quite relevant
Re: Show HN: I put PubMed in a vector DB
#19Earlier quoted context omitted.
We use pinecone and it is not ideal, looking at https://turbopuffer.com/ now. They look quite promising :)
Did you compare pinecone against pgvector with Postgres? Self hosted of course
Isn't it funny how the best Choice somehow always comes back to Postgres in the end XD (for most)