Long Read: Lessons from Building Semantic Search for GitHub and Why I Failed

Thanks for sharing! Do you have more details to share, e.g. did you just have a vector db, or did you have a main db as well?

In my research, Qdrant was also the top contender and I even created an account with them, but the need to sync two dbs put me off

> And overall, the fewer people use CF (or another provider of their size) the better.

I understand your sentiment, but I vehemently disagree.

The cloud provider space has rapidly become an oligopoly and CloudFlare is one of the few new entrants that (1) has sufficient scale to compete with the incumbents; (2) has new ideas that the incumbents cannot easily match (region earth, durable objects etc.).

For most production workloads, I would not even consider the newer cloud providers, but I sincerely meant it when I said I hope Cloudflare will succeed. They've also been very responsive to the feedback raised in the blogpost when I DM-ed them.

(On a side note re: difficulty for newcomers in this market, I used to be part of a team that would run e.g. staging and testing environments on a new serverless db provider, but would run prod on AWS Aurora. In retrospect, this did not make much sense either as you want your environments to be as similar as possible, which means new cloud providers have an even tougher time getting started.)

I am using a vps and it is dead simple and cheap. If my projects actually gained traction , switching from vps to a more scalable infra is not a big challenge. The biggest challenge is to find PMV as fast and as efficient as possible.

I was the first employee at a company which uses RAG (Halcyon), and I’ve been working through issues with various vector store providers for almost two years now. We’ve gone from tens of thousands to billions of embeddings in that timeframe - so I feel qualified to at least offer my opinion on the problem.

I agree that starting with pgvector is wise. It’s the thing you already have (postgres), and it works pretty well out of the box. But there are definitely gotchas that don’t usually get mentioned. Although the pgvector filtering story is better than it was a year ago, high-cardinality filters still feel like a bit of an afterthought (low-cardinality filters can be solved with partial indices even at scale). You should also be aware that the workload for ANN is pretty different from normal web-app stuff, so you probably want your embeddings in a separate, differently-optimized database. And if you do lots of updates or deletes, you’ll need to make sure autovacuum is properly tuned or else index performance will suffer. Finally, building HNSW indices in Postgres is still extremely slow (even with parallel index builds), so it is difficult to experiment with index hyperparameters at scale.

Dedicated vector stores often solve some of these problems but create others. Index builds are often much faster, and you’re working at a higher level (for better or worse) so there’s less time spent on tuning indices or database configurations. But (as mentioned in other comments) keeping your data in sync is a huge issue. Even if updates and deletes aren’t a big part of your workload, figuring out what metadata to index alongside your vectors can be challenging. Adding new pieces of metadata may involve rebuilding the entire index, so you need a robust way to move terabytes of data reasonably quickly. The other challenge I’ve found is that filtering is often the “special sauce” that vector store providers bring to the table, so it’s pretty difficult to reason about the performance and recall of various types of filters.

Hi Warren, great article. Would love to connect on what we're doing (also in Singapore). Please drop me a message gregor@vand.hk