Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we replaced Elasticsearch and MongoDB with Rust and RocksDB

radar.com

Post reply on HN
Print view View original

81–90 of 104 posts

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#81

Post by Aaargh20318 »

nickm12 wrote: post #63
pianoben wrote: post #60

Lol I "love" that the first benefit this company lists in their jobs page is "In-Office Culture". Do people actually believe that having to commute is a benefit ?

You can't reduce the in-office or remote experience purely to commuting. It's just one aspect about how and where you work and work life balance in general. But since you asked, yes, I actually enjoy commuting when it is less than 30 minutes each way and especially when it involves physical activities. My best commutes have been walking and biking commutes of around 20-25 minutes each way. They give me exercise, a ch…

> provide "space" between work and home

I don't get this. This idea that 'work life balance' should mean that the two should be compartmentalised to specific blocks of time seems counterproductive to me. To me it feels like an unnatural way of living. 8 hours in which I should only focus on work, 8 hours I should focus on everything else followed by 8 hours of sleep. I don't think that is how we are supposed to operate. Even the 8 hours of sleep in one block is not natural and a recent invention. Before industrialisation people used to sleep in multiple blocks (wikipedia: polyphasic sleeping)

The idea that you have to be 'on' for 8 hours at a time seems extremely stressful to me. No wonder you need an hour afterwards just to unwind. Interleaving blocks of work and personal time over the day feels much more natural and less stressful to me. WFH makes this possible. If I'm stuck on something, I can do something else for a while, maybe even take a short nap. The ability to focus and do mentally straining work comes in waves for me. Being able to go with my natural flow makes me both happier, more relaxed and more productive.

The key to work/life balance to me is not stricter separation but instead better integration.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#82

Post by LtWorf »

PepsiCola wrote: post #78
LtWorf wrote: post #64

Earlier quoted context omitted.

[flagged]

Or maybe they don't live in US where everything is by car :)

I live in sweden. I assure you public transport commute is no joke.

edit: and if you live outside of the city you'll need a car anyway.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#83

Post by brunohaid »

porridgeraisin wrote: post #76
jjordan wrote: post #18

Earlier quoted context omitted.

Typesense is an absolute beast, and it has a pretty great dev experience to boot.

Can you share what makes it better than competitors? And what's great about the dev experience? Did you use their cloud offering? The marketing material looks great, but I want to hear a user's experience.

For me it's a combination 1) solid foundational choices all along, no bolted on vanity features or constant rewrites chasing the latest trend, with everything well documented and 2) incredibly responsive founding team, so you get very quick answers from the people actually building it.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#84

Post by steve_adams_86 »

brunohaid wrote: post #69
atombender wrote: post #42

Earlier quoted context omitted.

DuckDB does not have any kind of sharding or clustering? It doesn't even have a server (unless you count the HTTP Server Extension)?

Good point and was mostly re Typesense (can't edit the comment anymore). But given that duckdb handles "take this n GB parquet file/shard from a random location, load it into memory and be ready in We use it for both the importer pipeline that processes the 2B row / 200GB compressed GBIF.org parquet dataset and queries like https://www.meso.cloud/plants/pinophyta/cupressales/pinopsid... and the sheer amount of functi…

Duckdb is so absurdly portable you can solve all kinds of performance and scaling concerns this way. It's really a wonderful, fun piece of software.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#85

Post by wavemode »

pm90 wrote: post #7

Slightly meta, but I find its a good sign that we're back to designing and blogging about in-house data storage systems/ Query engines again. There was an explosion of these in the 2010's which seemed to slow down/refocus on AI recently.

NoSQL/alternative databases became kind of a meme once people realized that 95% of enterprises can do fine with just Postgres.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#86

Post by mdaniel »

whakim wrote: post #57
cfors wrote: post #17

Earlier quoted context omitted.

I don't disagree that rock solid is a good choice, but there is a ton of innovation necessary for data stores. Especially in the context of embedding search, which this article is also trying to do. We need database that can efficiently store/query high-dimensional embeddings, and handle the nuance of real-world applications as well such as filtered-ANN. There is a ton of innovation in this space and it's crucial to…

I do not think data stores are a bottleneck for serving embedding search. I think the raft of new-fangled vector db services (or pgvector or whatever) can be a bottleneck because they are mostly optimized around the long tail of pretty small data. Real internet-scale search systems like ES or Vespa won’t struggle with serving embedding search assuming you have the necessary scale and time/money to invest in them.

> Real internet-scale search systems like ES

Oh, then you must have the secret sauce that allows scaling ES vector search beyond 10,000 results without requiring infinite RAM. I know their forums would welcome it, because that question comes up a lot

Or I guess that's why you included the qualifier about money to invest

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#87

Post by hyc_symas »

nekitamo wrote: post #56

I've used RocksDB a lot in the past and am very satisfied with it. It was helpful building a large write-heavy index where most of the data had to be compressed on disk. I'm wondering if anyone here has experience with LMDB and can comment on how they compare? https://www.symas.com/mdb I'm looking at it next for a project which has to cache and serve relatively small static data, and write and look up millions of ind…

LMDB is for read-heavy workloads. The opposite of RocksDB.

RocksDB can use thousands of file descriptors at once, on larger DBs. Makes it unsuitable for servers that may also need to manage thousands of client connections at once.

LMDB uses 2 file descriptors at most; just 1 if you don't use its lock management, or if you're serving static data from a readonly filesystem.

RocksDB requires extensive configuration to tune properly. LMDB doesn't require any tuning.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#88

Post by LtWorf »

throw738338 wrote: post #67
LtWorf wrote: post #65

Earlier quoted context omitted.

My office has some nice perks! 1. It's extremely cold and dark! I must wear extra clothes when going inside and I get depressed at wasting a day of nice weather in what looks like a WW1 bunker. 2. Terrible accessibility for disabled people! (such as myself) 3. Filthy toilets! 4. Internet is slower than at home! 5. Half the team lives somewhere else so all meetings are on teams anyway! 6. They couldn't afford a decent…

Friend works at office that allows dogs. Her workplace is one big dog toilet! She is expected to clean it (she is not toilet cleaner). She get sexually assaulted, when her boss shoved his dog into her crotch! There were some hospitalisations from work related injuries... Regular bullying, threats of violence.... Lovely office culture!

I did quit a job within 2 weeks because of casual racism and sex harassment in the office. But I was lucky to find something else that fast, to be able to do it.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#89

Post by commandersaki »

Aaargh20318 wrote: post #81
nickm12 wrote: post #63

Earlier quoted context omitted.

You can't reduce the in-office or remote experience purely to commuting. It's just one aspect about how and where you work and work life balance in general. But since you asked, yes, I actually enjoy commuting when it is less than 30 minutes each way and especially when it involves physical activities. My best commutes have been walking and biking commutes of around 20-25 minutes each way. They give me exercise, a ch…

> provide "space" between work and home I don't get this. This idea that 'work life balance' should mean that the two should be compartmentalised to specific blocks of time seems counterproductive to me. To me it feels like an unnatural way of living. 8 hours in which I should only focus on work, 8 hours I should focus on everything else followed by 8 hours of sleep. I don't think that is how we are supposed to opera…

Counterpoint: when I "wfh" I end up just sleeping 90% of my work hours and smashing out actual work for the remainder. When I'm in an office I'm productive 70% of my hours and it has nothing to do with accountability, just a proper office environment (and yes I have a work area at home). Regardless of going to office or wfh, I don't have set hours.

The overarching point is everyone is different, ymmv.

Re: How we replaced Elasticsearch and MongoDB with Rust and RocksDB

#90

Post by ellenhp »

mips_avatar wrote: post #59
ellenhp wrote: post #33

Earlier quoted context omitted.

There are a few piece of this that rely on proprietary data, especially the FastText training step, so that's a dead-end unfortunately (would love to be proven wrong). I'd consider subbing in a small bert model with a classifier head for something FOSS without access to tons of user data, but then you lose the ability to serve high qps.

I guess not having that would only breaking forward geocoding from an address?

My guess is that they're using FastText for semantic search, so it's more likely to break queries like "coffee near me" than address search, the latter likely being handled by tantivy. For context, I've also written a geocoder [0] based on tantivy. :)

[0] https://github.com/ellenhp/airmail

Post reply on HN