Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We use RethinkDB

blog.workshape.io

Post reply on HN
Print view View original

41–50 of 78 posts

Re: We use RethinkDB

#41

Post by neumino »

mbrock wrote: post #12
lewisjoe wrote: post #4

I've always wondered what's the best way to integrate a database engine with the application. 1) Use a middleware/ORM/Whatever which abstracts away the query-lang of the db, and provides a pluggable multi-db support 2) Just use native db query language with all exclusive features of the engine. Companies like workshape.io, why do they prefer the latter?

Abstractions are leaky. [1] ORM systems are highly complicated abstractions. Most of your developers will use them without understanding how they work. In many cases, the only way to understand how they work is to read the source code. They have magic features that are advertised as convenient but when they inevitably do something you don't want them to do you'll tear your hair out trying to circumvent them. They put…

You should check out thinky.

Because of the chainable query language you don't have to build a new abstractions. You can just use the same.

Meaning that there is nothing (or very little) to learn yp switch from the driver to thinky.

Re: We use RethinkDB

#42

Post by [deleted] »

felipesabino wrote: post #6

From RethinkDB docks [1], I am still a bit confused how this locking system works for read/write and also a bit skeptical regarding their claim that 'in most cases writes can be performed essentially lock-free'. I am using MongoDB and didn't have many issues when my databases had 120,000 documents either, the problem began when we hit the millions... The combination of write locks and our need for dynamic queries (me…

[deleted]

Re: We use RethinkDB

#43

Post by [deleted] »

felipesabino wrote: post #6

From RethinkDB docks [1], I am still a bit confused how this locking system works for read/write and also a bit skeptical regarding their claim that 'in most cases writes can be performed essentially lock-free'. I am using MongoDB and didn't have many issues when my databases had 120,000 documents either, the problem began when we hit the millions... The combination of write locks and our need for dynamic queries (me…

[deleted]

Re: We use RethinkDB

#44

Post by felipesabino »

lucian1900 wrote: post #40
felipesabino wrote: post #6

From RethinkDB docks [1], I am still a bit confused how this locking system works for read/write and also a bit skeptical regarding their claim that 'in most cases writes can be performed essentially lock-free'. I am using MongoDB and didn't have many issues when my databases had 120,000 documents either, the problem began when we hit the millions... The combination of write locks and our need for dynamic queries (me…

No database can be fast without indexes. If the queries you get are defined by your users, you can still create indexes for the common ones.

The way business logic in the system is currently designed , there are no 'common ones', except for some ids (already indexed) used in other process rather than this dynamic filter.

Based on that we also evaluated the approach of using something else like druid [1] [2] that is built for reading performance, but I am still studying possibilities and have no idea about the impact and problems a change like that would impose.

[1] http://druid.io/

[2] https://metamarkets.com/2014/building-a-data-pipeline-that-h...

Re: We use RethinkDB

#45

Post by mrkurt »

coffeemug wrote: post #32
jefficient wrote: post #29

Earlier quoted context omitted.

Are unique indexes planned for any of the upcoming releases?

The primary index for the table is always unique. We do not plan to implement secondary unique indexes. The philosophy behind RethinkDB is that if a feature cannot be efficiently scaled across multiple nodes we don't add it, and unfortunately unique secondary indexes are one such feature. AFAIK most NoSQL databases don't implement it, and a few that do take two approaches -- forbid it on multiple shards, or just take…

This is good, for what it's worth. Having different feature sets for sharded vs unsharded is just utterly confusing, and something MongoDB got really wrong.

Re: We use RethinkDB

#46

Post by simonpantzare »

We have used RethinkDB in production for a handful of months now. 100M docs, 250 GB data spread out on two servers.

We added it to the mix because it got increasingly difficult to tune SQL queries involved in building API responses, especially for endpoints that needed to pull data from many tables.

Our limited experience of MySQL operations was also a factor. We're on 5.5 and couldn't do some table operations that seemed promising without service disruptions. There were solutions to perform the actions we wanted without downtime but they scared us a bit. We also looked into upgrading to 5.6 or MariaDB but that seemed like it would take a long time and need much testing, while there were no guarantees that we would see performance gains.

We looked for alternative solutions and found RethinkDB. We reused the parts that serialize data for the API and put the resulting documents in RethinkDB. Then we had our API request handlers pull data from there instead of from MySQL and added indexes to support various kinds of filtering, pagination, and so on. We built this for our most problematic endpoint and got the two-server cluster up and running in about a week, tried it out on employees for another week, and then enabled it for everyone (with the option to quickly fall back to pulling data from MySQL).

This turned out to work well and we saw good response times, so we did the same thing for other endpoints.

There's some complexity involved in keeping RethinkDB docs up to date with MySQL (where writes still go) but nothing extreme and we haven't had many sync issues.

RethinkDB has been rock solid and it's a joy to operate.

Re: We use RethinkDB

#47

Post by tracker1 »

felipesabino wrote: post #33
tracker1 wrote: post #14

Earlier quoted context omitted.

FYI, with MongoDB, just because you can't and shouldn't index everything, doesn't mean you can't have any indexes... if your most common fields bring your queries down, they're still pretty helpful. I actually really like where RethinkDB is headed, and within the year most of my issues should be resolved. Another couple databases to consider, depending on your needs would be ElasticSearch and Cassandra... it reallly…

Oh yes, agreed! and we do have some indexes, but unfortunately it isn't enough :( And we are actually considering ElasticSearch to deal with this querying performance issue.

We had a similar issue with queries that didn't match an index... later pages would time out, etc... we limited our results for that class of queries... The next generation will use cassandra with some custom searching/caching... that will work a bit differently.

Re: We use RethinkDB

#48

Post by JulianMorrison »

coffeemug wrote: post #35
pquerna wrote: post #31

Is anyone using RethinkDB as a "lightweight" Business Intelligence / Analytics / Datawharehouse store? (Maybe for use cases like Amazon Redshift?) It seems like there could be a sweet spot of their nice query language, schemaless, and easy scaling for moderate size data sets? I'm kinda tired of having to go all in on a big hadoop-ecosystem just to figure out average X to Y in a dataset....

Slava @ RethinkDB here. Some of our biggest users (to be announced in a few weeks for 2.0) use RethinkDB this way. You can't really do deep analytics/machine learning as RethinkDB wasn't designed for that, but if you want to store a lot of data, and then run lightweight aggregation or map-reduce queries on that data, Rethink turns out to be a really good product for it. One issue I see with this path is that if your…

That sounds like a job for the changes feed, pre-digest data with a query and then pipe its changes feed into Hadoop's storage. (How fast can change feeds run? Would that end up being a bottleneck?)

Re: We use RethinkDB

#49

Post by TylerE »

dubcanada wrote: post #11
emergentcypher wrote: post #5

Why is it so trendy these days to say that everything was built "with love"?

Would you much rather it was not built with love? I really fail to see how saying "built with love" matters at all.

I would kinda like a DB built with cold-heated ruthless efficiency and precision, actually.

Re: We use RethinkDB

#50

Post by RealGeek »

simonpantzare wrote: post #46

We have used RethinkDB in production for a handful of months now. 100M docs, 250 GB data spread out on two servers. We added it to the mix because it got increasingly difficult to tune SQL queries involved in building API responses, especially for endpoints that needed to pull data from many tables. Our limited experience of MySQL operations was also a factor. We're on 5.5 and couldn't do some table operations that s…

What is the hardware of your servers and how many queries are you performing per second? How is the performance and latency?
Post reply on HN