Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we replaced Elasticsearch and MongoDB with Rust and RocksDB

radar.com

Post reply on HN
Print view View original

31–40 of 104 posts

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#31

Post by unsuitable »

mikeocool wrote: post #8
softwaredoug wrote: post #6

It’s interesting as someone in the search space how many companies are aiming to “replace Elasticsearch”

In my experience, the care and feeding that goes into an Elastic Search cluster feels like it's often substantially higher than that involved in the primary data store, which has always struck me as a little odd (particularly in cases where the primary data store is an RDBMS). I'd be very happy to use simpler more bulletproof solutions with a subset of ES's features for different use cases.

In my experience Elastic Search lacks fundamental tooling, like a CLI that copies data between nodes.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#32

Post by ellenhp »

cicloid wrote: post #13
pbowyer wrote: post #4

Earlier quoted context omitted.

Doesn't sound like it, but it's a nice writeup of the tools they stitched together. For someone to copy and open source... hopefully :)

Tempted, specially for switching H3 instead of S2… I prototyped a similar solution a couple of weeks ago, so I could probably do a second pass

What's wrong with S2? H3 is so much more complex for very little gain from what I can tell.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#33

Post by ellenhp »

pbowyer wrote: post #4
sophia01 wrote: post #3

They're not open sourcing it though?

Doesn't sound like it, but it's a nice writeup of the tools they stitched together. For someone to copy and open source... hopefully :)

There are a few piece of this that rely on proprietary data, especially the FastText training step, so that's a dead-end unfortunately (would love to be proven wrong). I'd consider subbing in a small bert model with a classifier head for something FOSS without access to tons of user data, but then you lose the ability to serve high qps.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#34

Post by tracker1 »

Nice... it's cool to see how different companies are putting together best fit solutions. I'm also glad that they at least started out with off the shelf apps instead of jumping to something like a bespoke solution early on.

Quickwit[1] looks interesting, found via Tantivity reference. Kind of like ES w/ Lucene.

1. https://github.com/quickwit-oss/quickwit

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#35

Post by heipei »

dewey wrote: post #16
mikeocool wrote: post #8

Earlier quoted context omitted.

In my experience, the care and feeding that goes into an Elastic Search cluster feels like it's often substantially higher than that involved in the primary data store, which has always struck me as a little odd (particularly in cases where the primary data store is an RDBMS). I'd be very happy to use simpler more bulletproof solutions with a subset of ES's features for different use cases.

To add another data point: After working with ES for the past 10 years in production I have to say that ES is never giving us any headaches. We've had issues with ScyllaDB, Redis etc. but ES is just chugging along and just works. The one issue I remember is: On ES 5 we once had an issue early on where it regularly went down, turns out that some _very long_ input was being passed into the search by some scraper and ki…

I agree, and I don't get where the claims that ES is hard to operate originate from. Yeah, if you allow arbitrary aggregations that exceed the heap space, or if you allow expensive queries that effectively iterate over everything you're gonna have a bad time. But apart from those, as long as you understand your data model, your searches and how data is indexed, ES is absolutely rock-solid, scales and performs like a beast. We run a 35-node cluster with ~ 240TB of disk, 4.5TB of RAM, and about 100TB of documents and are able to serve hundreds of queries. The whole thing does not require any maintenance apart from replacing nodes that failed from unrelated causes (hardware, hosting). Version upgrades are smooth as well.

The only bigger issue we had was when we initially added 10 nodes to double the initial capacity of the cluster. Performance tanked as a result, and it took us about half a day until we finally figured out that the new nodes were using dmraid (Linux RAID0) and as a result the block devices had a really high default read-ahead value (8192) compared to the existing nodes, which resulted in heavy read amplification. The ES manual specifically documents this, but since we hadn't run into this issue ourselves it took us a while to realise what was at fault.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#36

Post by ericcholis »

brunohaid wrote: post #14

Bit thin on details and not looking like they’ll open source it, but if someone clicked the post because they’re looking for their “replace ES” thing: Both https://typesense.org/ and https://duckdb.org/ (with their spatial plugin) are excellent geo performance wise, the latter now seems really production ready, especially when the data doesn’t change that often. Both fully open source including clustered/sharded setu…

Typsense as a product has been great (hosted cluster). Customer support has been awesome as well.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#37

Post by mcdonje »

brunohaid wrote: post #14

Bit thin on details and not looking like they’ll open source it, but if someone clicked the post because they’re looking for their “replace ES” thing: Both https://typesense.org/ and https://duckdb.org/ (with their spatial plugin) are excellent geo performance wise, the latter now seems really production ready, especially when the data doesn’t change that often. Both fully open source including clustered/sharded setu…

Not sure what they'll opensource. The rust code? They're calling it a DB, but they described an entire stack.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#39

Post by francoismassot »

tracker1 wrote: post #34

Nice... it's cool to see how different companies are putting together best fit solutions. I'm also glad that they at least started out with off the shelf apps instead of jumping to something like a bespoke solution early on. Quickwit[1] looks interesting, found via Tantivity reference. Kind of like ES w/ Lucene. 1. https://github.com/quickwit-oss/quickwit

it's tantivy :)

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#40

Post by nchmy »

mikeocool wrote: post #8
softwaredoug wrote: post #6

It’s interesting as someone in the search space how many companies are aiming to “replace Elasticsearch”

In my experience, the care and feeding that goes into an Elastic Search cluster feels like it's often substantially higher than that involved in the primary data store, which has always struck me as a little odd (particularly in cases where the primary data store is an RDBMS). I'd be very happy to use simpler more bulletproof solutions with a subset of ES's features for different use cases.

Check out manticoresearch - it's older than Lucene (which elasticsearch is built on), faster and simpler.
Post reply on HN