Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RethinkDB: An open-source distributed database built with love over three years

rethinkdb.com

Post reply on HN
Print view View original

41–50 of 247 posts

Re: RethinkDB: An open-source distributed database built with love over three years

#41

Post by coffeemug »

eknkc wrote: post #40

So, no secondary indexes right? Will doing a query like "age > 25" perform something equivalent to a full table scan?

Yes, there are no secondary indexes right now (they're coming, but it will take a little time). Currently "age > 25" will do a scan, but if the db is sharded it will touch only the relevant shards and the query will be completely parallelized. We also do internal sharding for multicore performance, and the query will be parallelized across cores as well.

All of that is only good for relatively small amounts of data though, so we'll be adding secondary indexes soon.

Re: RethinkDB: An open-source distributed database built with love over three years

#42

Post by jdoliner »

eknkc wrote: post #40

So, no secondary indexes right? Will doing a query like "age > 25" perform something equivalent to a full table scan?

Correct currently no secondary indices. Ranges on fields other than the primary key will require a full table scan.

-Joe Doliner, engineer at RethinkDB

Re: RethinkDB: An open-source distributed database built with love over three years

#43

Post by jamesli »

Great work! One question: is there any manual that explains the implementation details of the internals? Some manual similar to those Oracle, MySQL, Postgres, etc. provide?

The only docs I found in the company website that goes deep into the internals are Advanced FAQ (http://www.rethinkdb.com/docs/advanced-faq/). It is more of an architecture view, though.

The reason I ask is that with a good understanding on the internals, the engineers who understand database internals and distributed systems will have an "more" accurate idea on the capabilities and the limits of the features. Thus, if they decide to adopt RethinkDB, the understanding will help them design their applications to take advantages of the benefits and avoid the potential issues (or surprises!). MongoDB was not very good at documentation. It claims this or that feature works smoothly. Then, people found out many potential issues and limitations. That is one reason it leaves a bad tastes to many engineers.

Re: RethinkDB: An open-source distributed database built with love over three years

#44

Post by mej10 »

Goopplesoft wrote: post #24
mej10 wrote: post #21

Earlier quoted context omitted.

What do you see as the potential areas where RethinkDB will shine? Also, I am excited to try this out. I always enjoyed your writings and I am sure you + team have made something awesome.

Hah that question is almost word for word in the faq: http://www.rethinkdb.com/docs/faq/

Yep, you got me. I hadn't read the FAQ. Thanks for that.

Re: RethinkDB: An open-source distributed database built with love over three years

#45

Post by mej10 »

jdoliner wrote: post #32
mej10 wrote: post #21

Earlier quoted context omitted.

What do you see as the potential areas where RethinkDB will shine? Also, I am excited to try this out. I always enjoyed your writings and I am sure you + team have made something awesome.

Joe Doliner - Engineer at RethinkDB here. RethinkDB is designed for small teams with big data challenges. When you're just starting up a new project ideally you want to just boot your database up and start throwing data at it without worrying about schema. However with other products on the market, most notably Mongo, there are a lot of features that stop working when you get to a large scale. We've been very careful…

Thanks Joe.

Re: RethinkDB: An open-source distributed database built with love over three years

#46

Post by jacquesm »

tjic wrote: post #39

What the heck does "built with love" even mean? Is this just a hipster marketing term to tell us that it's small and cute and made by people who play ukuleles and ride unicycles in their spare time, and not by evil corporate people who commute to work and have mortgages? I find a lot of advertising eyeroll inducing, and the current trend of more-hipster-than-thou posturing is right at the top.

Releasing a project like this means working very hard for a long time without anybody patting you on the back saying you're doing a good job. It's exhausting, a labour of love.

Get it now?

Re: RethinkDB: An open-source distributed database built with love over three years

#47

Post by coffeemug »

jamesli wrote: post #43

Great work! One question: is there any manual that explains the implementation details of the internals? Some manual similar to those Oracle, MySQL, Postgres, etc. provide? The only docs I found in the company website that goes deep into the internals are Advanced FAQ ( http://www.rethinkdb.com/docs/advanced-faq/ ). It is more of an architecture view, though. The reason I ask is that with a good understanding on the…

There currently isn't, beyond the advanced FAQ. This isn't by design -- writing really good detailed architecture papers takes a lot of time, and we were 100% focused on getting the product out. We'll get much better at documenting the internals, but it will take some time.

Re: RethinkDB: An open-source distributed database built with love over three years

#48

Post by gruseom »

tjic wrote: post #39

What the heck does "built with love" even mean? Is this just a hipster marketing term to tell us that it's small and cute and made by people who play ukuleles and ride unicycles in their spare time, and not by evil corporate people who commute to work and have mortgages? I find a lot of advertising eyeroll inducing, and the current trend of more-hipster-than-thou posturing is right at the top.

You could not have gotten these guys more wrong. They are serious technologists who have been working day and night for years to build something that they deeply believe in. Every hacker's heart should be warmed by the fact that they kept at it.

When you have a vision of something great that ought to exist and set about bringing it into the world, you are in an isolated position: other people don't yet see what you see. This leads to a lot of doubt by others and by yourself too. The longer it takes, the more exposed you are. To make it through that you are going to need a deeper source of motivation – an underground spring. Love is a fine word for this, and it makes me happy that Slava put it in his title: it's a clue to this experience that rarely gets mentioned, especially in the land of pivots and MVPs and weekend hacks.

Re: RethinkDB: An open-source distributed database built with love over three years

#49

Post by Xorlev »

coffeemug wrote: post #37
continuations wrote: post #33

* In the previous incarnation of rethinkdb the focus was on maximizing performance on SSDs. Is this still the case - does rethinkDB perform better than other databases on SSDs? Do you have any benchmark numbers? * How does rethinkdb compare to MySQL Cluster? Both are distributed, replicated databases with a sql-like query language. * Any plan to offer a java client?

* The SSD-optimized storage engine is running under the clustering engine. I'm wary of saying 'better' or 'worse' in case of benchmarks, because they're really tricky to do right. We'll be publishing well-researched benchmarks as soon as we can, but it will take time. * RethinkDB has flexible schemas and a query language that integrates straight into the host programming language and doesn't require string interpolat…

I'm impressed that you're taking the time to do proper, well-researched benchmarks. They're really tough to get right. It really comes down to your own specific workload that matters anyways.

This feels new and refreshing, I hope things turn out positively for you.

Re: RethinkDB: An open-source distributed database built with love over three years

#50

Post by spicyj »

coffeemug wrote: post #41
eknkc wrote: post #40

So, no secondary indexes right? Will doing a query like "age > 25" perform something equivalent to a full table scan?

Yes, there are no secondary indexes right now (they're coming, but it will take a little time). Currently "age > 25" will do a scan, but if the db is sharded it will touch only the relevant shards and the query will be completely parallelized. We also do internal sharding for multicore performance, and the query will be parallelized across cores as well. All of that is only good for relatively small amounts of data t…

How does it know which shards are relevant? I thought that data was sharded by primary key.
Post reply on HN