Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we replaced Elasticsearch and MongoDB with Rust and RocksDB

radar.com

Post reply on HN
Print view View original

91–100 of 104 posts

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#91

Post by whakim »

mdaniel wrote: post #86
whakim wrote: post #57

Earlier quoted context omitted.

I do not think data stores are a bottleneck for serving embedding search. I think the raft of new-fangled vector db services (or pgvector or whatever) can be a bottleneck because they are mostly optimized around the long tail of pretty small data. Real internet-scale search systems like ES or Vespa won’t struggle with serving embedding search assuming you have the necessary scale and time/money to invest in them.

> Real internet-scale search systems like ES Oh, then you must have the secret sauce that allows scaling ES vector search beyond 10,000 results without requiring infinite RAM. I know their forums would welcome it, because that question comes up a lot Or I guess that's why you included the qualifier about money to invest

Would you mind putting aside the snark? I have a couple questions. How large is the corpus? I am also curious about the use-case for top-k ANN, k > 10000?

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#92

Post by farsa »

whakim wrote: post #91
mdaniel wrote: post #86

Earlier quoted context omitted.

> Real internet-scale search systems like ES Oh, then you must have the secret sauce that allows scaling ES vector search beyond 10,000 results without requiring infinite RAM. I know their forums would welcome it, because that question comes up a lot Or I guess that's why you included the qualifier about money to invest

Would you mind putting aside the snark? I have a couple questions. How large is the corpus? I am also curious about the use-case for top- k ANN, k > 10000?

Not the person you have asked but at work (we are a CRM platform) we allow our clients to arbitrarily query their userbase to find matching users for marketing campaigns (email, sms, whatsapp). These campaigns can some times target a few hundred thousand people. We are on a really ancient version of ES, but it sucks at this job in terms of throughput. Some experimenting with bigquery indicates it is so much better at mass exporting.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#93

Post by farsa »

benjiro wrote: post #70
j_kao wrote: post #19

Earlier quoted context omitted.

Author here! We were really motivated to turn a "distributed system" problem into a "monolithic system" from an operations perspective and felt this was achievable with current hardware, which is why we went with in-process, embedded storage systems like RocksDB and Tantivy. Memory-mapping lets us get pretty far, even with global coverage. We are always able to add more RAM, especially since we're running in the clou…

Why not just use a open source solution like paradedb ... . Paradedb = postgres pg_search plugin (the base is tantivy). Need anything else like vectors or whatever, get the plugins for postgres. The only thing your missing is a LSM solution like RocksDB. See Orioledb what is supposed to become a plugin storage engine for postgres but not yet out of beta. Feels like people reinvent the wheel very often.

What was your experience like putting such thing together?

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#94

Post by whakim »

farsa wrote: post #92
whakim wrote: post #91

Earlier quoted context omitted.

Would you mind putting aside the snark? I have a couple questions. How large is the corpus? I am also curious about the use-case for top- k ANN, k > 10000?

Not the person you have asked but at work (we are a CRM platform) we allow our clients to arbitrarily query their userbase to find matching users for marketing campaigns (email, sms, whatsapp). These campaigns can some times target a few hundred thousand people. We are on a really ancient version of ES, but it sucks at this job in terms of throughput. Some experimenting with bigquery indicates it is so much better at…

Fair; my question was mostly in the context of ANN, since that was the discussion point - I have to assume ES (as a search engine) would not necessarily be the right tool for data warehousing types of workloads.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#95

Post by nickm12 »

Aaargh20318 wrote: post #81
nickm12 wrote: post #63

Earlier quoted context omitted.

You can't reduce the in-office or remote experience purely to commuting. It's just one aspect about how and where you work and work life balance in general. But since you asked, yes, I actually enjoy commuting when it is less than 30 minutes each way and especially when it involves physical activities. My best commutes have been walking and biking commutes of around 20-25 minutes each way. They give me exercise, a ch…

> provide "space" between work and home I don't get this. This idea that 'work life balance' should mean that the two should be compartmentalised to specific blocks of time seems counterproductive to me. To me it feels like an unnatural way of living. 8 hours in which I should only focus on work, 8 hours I should focus on everything else followed by 8 hours of sleep. I don't think that is how we are supposed to opera…

> This idea that 'work life balance' should mean that the two should be compartmentalised to specific blocks of time seems counterproductive to me.

Different people are different and can have different preferences.

For me, having different physical spaces helps me focus on work at work and my family at home. When they are the same physical space, both suffer. I'm not saying everyone should feel this way.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#96

Post by smsm42 »

nickm12 wrote: post #63
pianoben wrote: post #60

Lol I "love" that the first benefit this company lists in their jobs page is "In-Office Culture". Do people actually believe that having to commute is a benefit ?

You can't reduce the in-office or remote experience purely to commuting. It's just one aspect about how and where you work and work life balance in general. But since you asked, yes, I actually enjoy commuting when it is less than 30 minutes each way and especially when it involves physical activities. My best commutes have been walking and biking commutes of around 20-25 minutes each way. They give me exercise, a ch…

> provide "space" between work and home.

This is part of the company culture. If the company respects the boundary between work and personal life, and it's a cultural value, then it shouldn't be a problem for you establishing a space even without going to the office. You just close down your work laptop, put it aside and open it up next time when it's time to work again. Of course, there's stuff like on-call shifts, and there's a temptation to just stay later and finish this one thing, but if the company culture does not expect you to be tethered to work 24x7 then it's doable. If the culture is right, you don't need a physical barrier for this to be doable.

> so I started taking hour long walks at the end of every day to reset. It helped a lot.

A good habit. I dont see why any remote worker couldn't do that.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#97

Post by kosolam »

vips7L wrote: post #53
kosolam wrote: post #15

Side note 1: ES can also be embedded in your app (on the JVM). Note 2: I actually used RocksDB to solve many use cases and it’s quite powerful and very performant. If anything from this post take this, it’s open source and a very solid building block. Note 3: I would like to test drive quickwit as an ES replacement. Haven’t got the time yet.

I really enjoy embedding things in the vm. I run a discord bot with a few thousand users with embedded H2. Recently I’ve been looking at trying to embed keycloak (or something similar) for some other apps.

I did that with ES to squeeze performance but IIRC it didn’t really produce meaningful results. Otherwise, for most use cases an integration is better imho than embedding stuff that is when you have a full software service such as keycloack or ES. Rocksdb and h2 are tailor made as embedded libraries

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#98

Post by cfors »

whakim wrote: post #57
cfors wrote: post #17

Earlier quoted context omitted.

I don't disagree that rock solid is a good choice, but there is a ton of innovation necessary for data stores. Especially in the context of embedding search, which this article is also trying to do. We need database that can efficiently store/query high-dimensional embeddings, and handle the nuance of real-world applications as well such as filtered-ANN. There is a ton of innovation in this space and it's crucial to…

I do not think data stores are a bottleneck for serving embedding search. I think the raft of new-fangled vector db services (or pgvector or whatever) can be a bottleneck because they are mostly optimized around the long tail of pretty small data. Real internet-scale search systems like ES or Vespa won’t struggle with serving embedding search assuming you have the necessary scale and time/money to invest in them.

Sure they can handle the basic case of ANN. But ANN still doesn’t have good stories for lots of real-world problems.

* filterable ANN, decomposes into prefiltering or postfiltering.

* dynamic updates and versioning is still very difficult

* slow building of graph indexes

* adding other signals into the search, such as query time boosting for recent docs.

I don’t disagree these systems can work but innovation is still necessary. We are not in a “data stores are solved” world.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#99

Post by nickm12 »

smsm42 wrote: post #96
nickm12 wrote: post #63

Earlier quoted context omitted.

You can't reduce the in-office or remote experience purely to commuting. It's just one aspect about how and where you work and work life balance in general. But since you asked, yes, I actually enjoy commuting when it is less than 30 minutes each way and especially when it involves physical activities. My best commutes have been walking and biking commutes of around 20-25 minutes each way. They give me exercise, a ch…

> provide "space" between work and home. This is part of the company culture. If the company respects the boundary between work and personal life, and it's a cultural value, then it shouldn't be a problem for you establishing a space even without going to the office. You just close down your work laptop, put it aside and open it up next time when it's time to work again. Of course, there's stuff like on-call shifts,…

> it shouldn't be a problem for you establishing a space even without going to the office.

No, this was nothing to do with company culture. This was just my own mental response to just always being at home. Admittedly, the pandemic accentuated this because we weren't going anywhere even on weekends and evenings. But even as things opened up and we resumed our normal socialization, I returned to the office long before most people because I needed the mental and physical distance.

I know I'm atypical. In those early days,I estimated fewer than 5% of people in my office were voluntarily returning and even today when we're at RTO 3 days a week, most people do exactly that and no more.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#100

Post by whakim »

cfors wrote: post #98
whakim wrote: post #57

Earlier quoted context omitted.

I do not think data stores are a bottleneck for serving embedding search. I think the raft of new-fangled vector db services (or pgvector or whatever) can be a bottleneck because they are mostly optimized around the long tail of pretty small data. Real internet-scale search systems like ES or Vespa won’t struggle with serving embedding search assuming you have the necessary scale and time/money to invest in them.

Sure they can handle the basic case of ANN. But ANN still doesn’t have good stories for lots of real-world problems. * filterable ANN, decomposes into prefiltering or postfiltering. * dynamic updates and versioning is still very difficult * slow building of graph indexes * adding other signals into the search, such as query time boosting for recent docs. I don’t disagree these systems can work but innovation is still…

* Filterable ANN certainly decomposes into pre- and post-filtering, and there is definitely a lot of interesting innovation occurring around filterable ANN. But large-scale search systems currently do a pretty good job with pre-filtering, falling back to brute force search in the case of restrictive filters.

* You'd have to be a bit more exact re: dynamic updates/versioning for me to understand the challenges you're facing.

* Building graph indices can be slow, but in my experience (billions of embeddings) it is possible to build HNSW indices in tens of minutes.

* How is this any different to combining traditional keyword search with, say, recency boosting?

Post reply on HN