Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we replaced Elasticsearch and MongoDB with Rust and RocksDB

radar.com

Post reply on HN
Print view View original

51–60 of 104 posts

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#51

Post by Jweb_Guru »

0xbadcafebee wrote: post #48

Rocks is a fork of Level, and Level is well known for data corruption and other bugs. They are both "run at production scale", but at least back when I worked on stuff that used Level, nobody talked publicly about all the toil spent on cleaning up and repairing Level to keep the services based on it running. Whenever you see an advertisement like this (these posts are ads for the companies publishing them), they will…

RocksDB diverged from LevelDB a long time ago at this point and has had extensive work done on it by both industry and academia. It's not a toy database like LevelDB was. I can't speak to the problems they're supposedly hiding in their stack, but they are unlikely to come from RocksDB.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#52

Post by KAdot »

0xbadcafebee wrote: post #48

Rocks is a fork of Level, and Level is well known for data corruption and other bugs. They are both "run at production scale", but at least back when I worked on stuff that used Level, nobody talked publicly about all the toil spent on cleaning up and repairing Level to keep the services based on it running. Whenever you see an advertisement like this (these posts are ads for the companies publishing them), they will…

This is not my experience. I've been running RocksDB for 4 years on thousands of machines, each storing terabytes of data, and I haven't seen a single correctness issue caused by RocksDB.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#53

Post by vips7L »

kosolam wrote: post #15

Side note 1: ES can also be embedded in your app (on the JVM). Note 2: I actually used RocksDB to solve many use cases and it’s quite powerful and very performant. If anything from this post take this, it’s open source and a very solid building block. Note 3: I would like to test drive quickwit as an ES replacement. Haven’t got the time yet.

I really enjoy embedding things in the vm. I run a discord bot with a few thousand users with embedded H2. Recently I’ve been looking at trying to embed keycloak (or something similar) for some other apps.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#54

Post by chatmasta »

lisbbb wrote: post #47
binarymax wrote: post #29

Earlier quoted context omitted.

Lots of other options here if you don't like managing. You can use Elastic cloud, Bonsai.io, and others

A lot of places can't put their data just anywhere.

And they can pay the vendors for "bring your own cloud" or similar. If data sovereignty is important to them, then they can probably afford it. And if cost is an issue, then they wouldn't be looking at hosted solutions in the first place.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#55

Post by mdaniel »

nchmy wrote: post #40
mikeocool wrote: post #8

Earlier quoted context omitted.

In my experience, the care and feeding that goes into an Elastic Search cluster feels like it's often substantially higher than that involved in the primary data store, which has always struck me as a little odd (particularly in cases where the primary data store is an RDBMS). I'd be very happy to use simpler more bulletproof solutions with a subset of ES's features for different use cases.

Check out manticoresearch - it's older than Lucene (which elasticsearch is built on), faster and simpler.

GPLv3 https://github.com/manticoresoftware/manticoresearch/blob/13...

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#56

Post by nekitamo »

I've used RocksDB a lot in the past and am very satisfied with it. It was helpful building a large write-heavy index where most of the data had to be compressed on disk.

I'm wondering if anyone here has experience with LMDB and can comment on how they compare?

https://www.symas.com/mdb

I'm looking at it next for a project which has to cache and serve relatively small static data, and write and look up millions of individual points per minute.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#57

Post by whakim »

cfors wrote: post #17
8n4vidtmkvmk wrote: post #12

Earlier quoted context omitted.

Is it good? What's left to innovate on in this space? I don't really want experimental data stores. Give me something rock solid.

I don't disagree that rock solid is a good choice, but there is a ton of innovation necessary for data stores. Especially in the context of embedding search, which this article is also trying to do. We need database that can efficiently store/query high-dimensional embeddings, and handle the nuance of real-world applications as well such as filtered-ANN. There is a ton of innovation in this space and it's crucial to…

I do not think data stores are a bottleneck for serving embedding search. I think the raft of new-fangled vector db services (or pgvector or whatever) can be a bottleneck because they are mostly optimized around the long tail of pretty small data. Real internet-scale search systems like ES or Vespa won’t struggle with serving embedding search assuming you have the necessary scale and time/money to invest in them.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#58

Post by mips_avatar »

j_kao wrote: post #21
sophia01 wrote: post #3

They're not open sourcing it though?

It's a bit difficult at the moment, given we have a lot of proprietary data at the moment and a lot of the logic follows it. I'm hoping we can get it to a state where it can be indexed and serving OSM data but that is going to take some time. That being said, we are currently working on getting our Google S2 Rust bindings open-sourced. This is a geo-hashing library that makes it very easy to write a reverse geocoder,…

Could you write a photon replacement if you had that? I would love to spend less per month running photon for my project.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#59

Post by mips_avatar »

ellenhp wrote: post #33
pbowyer wrote: post #4

Earlier quoted context omitted.

Doesn't sound like it, but it's a nice writeup of the tools they stitched together. For someone to copy and open source... hopefully :)

There are a few piece of this that rely on proprietary data, especially the FastText training step, so that's a dead-end unfortunately (would love to be proven wrong). I'd consider subbing in a small bert model with a classifier head for something FOSS without access to tons of user data, but then you lose the ability to serve high qps.

I guess not having that would only breaking forward geocoding from an address?
Post reply on HN