Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RethinkDB: An open-source distributed database built with love over three years

rethinkdb.com

Post reply on HN
Print view View original

111–120 of 247 posts

Re: RethinkDB: An open-source distributed database built with love over three years

#111

Post by optymizer »

perfunctory wrote: post #61

Please stop using json as a data model. I have no idea how to represent dates, or timestamps, or colors, or any other unsupported data type.

So because you have no idea how to represent some arbitrary things with JSON, we should all stop using it? You, sir, are not making any sense.

Also, you seem to be confusing primitive data types with complex data types. Yes, JSON doesn't have a 'color' data type. But guess what? Neither does C, nor Java. If you want a 'color' type, you'll have to create one yourself! Mind blowing, I know.

So here, let me suggest a possible solution:

dates: string

timestamp: integer

colors: string (RGB,BGR,RGBA,...), integer, object

Part of 'data modeling' is to model your data out of basic types. Shocking! If JSON had types for every type of object under the Sun (like you seem to want), JSON parsers would be a lot more complicated and little thinking would be required in the process of modeling your data.

Re: RethinkDB: An open-source distributed database built with love over three years

#112

Post by optymizer »

perfunctory wrote: post #61

Please stop using json as a data model. I have no idea how to represent dates, or timestamps, or colors, or any other unsupported data type.

So because you have no idea how to represent some arbitrary things with JSON, we should all stop using it? You, sir, are not making any sense.

Also, you seem to be confusing primitive data types with complex data types. Yes, JSON doesn't have a 'color' data type. But guess what? Neither does C, nor Java. If you want a 'color' type, you'll have to create one yourself! Mind blowing, I know.

So here, let me suggest a possible solution:

dates: string timestamp: integer colors: string (RGB,BGR,RGBA,...), integer, object

Part of 'data modeling' is to model your data out of basic types. Shocking! If JSON had types for every type of object under the Sun (like you seem to want), JSON parsers would be a lot more complicated and little thinking would be required in the process of modeling your data.

Re: RethinkDB: An open-source distributed database built with love over three years

#113

Post by m0th87 »

How do filters work? They seem pretty difficult implementation-wise since you can write them in any of the language bindings. My first guess is that you pipe all the data in a table to the client, and the client itself does the filtration. But this would be extraordinarily inefficient.

Re: RethinkDB: An open-source distributed database built with love over three years

#114

Post by joevandyk »

tcwc wrote: post #63
joevandyk wrote: post #57

Why would you use this over PostgreSQL, especially with pg's new json support?

JSON support in Postgres is currently limited to a validated plain text field, it doesn't let you efficiently query inside the json object.

There's nothing stopping you from writing functions that can index json documents. plv8 makes this easy.

Re: RethinkDB: An open-source distributed database built with love over three years

#115

Post by old-gregg »

m0th87 wrote: post #113

How do filters work? They seem pretty difficult implementation-wise since you can write them in any of the language bindings. My first guess is that you pipe all the data in a table to the client, and the client itself does the filtration. But this would be extraordinarily inefficient.

Nope, they build an AST for filter expressions and compile it on the server IIRC. The client gets filtered data from the server.

Re: RethinkDB: An open-source distributed database built with love over three years

#116

Post by jdoliner »

m0th87 wrote: post #113

How do filters work? They seem pretty difficult implementation-wise since you can write them in any of the language bindings. My first guess is that you pipe all the data in a table to the client, and the client itself does the filtration. But this would be extraordinarily inefficient.

Piping all the data to the client would be extremely inefficient. Fortunately we don't do that.

When a filter is written in the client language it gets compiled into a protocol buffer which is sent to the cluster. This gets compiled into a query which is sent to each of the relevant shards for the table. This query has the filter baked right into it. The shards then go through their local copy of the data and filter out the rows which do not meet the query predicate. This data gets returned to the coordinating node and eventually to the user. Thus only the data the will actually be returned is ever transferred over the network.

Furthermore this process is done lazily. On the client side rather than getting back a huge array with the results of your filter you get back an iterator. This iterator stores a buffer of data which will be refilled as it is incremented.

Re: RethinkDB: An open-source distributed database built with love over three years

#117

Post by szopa »

jdoliner wrote: post #92
szopa wrote: post #67

Nice work! It seems that you are well aware of the tradeoffs that you are taking and communicating it openly in your documentation (and your choices seem to be very reasonable). I really like the tone of your communication – it seems essentially BS/koolaid free. 1. How much data can you put in one instance before seeing performance degradation? I know that you still working on good benchmarks – but do you have any ba…

Hi, here to answer question number 4. Short answer: Our configuration data is most similar to git. Any machine can be used as an administrative node via the WebUI or the CLI. It will make changes to the metadata which then get pushed to the other nodes. If 2 nodes make conflicting changes you get a conflict which the system will help you to merge. Long Answer Cluster configuration is stored in semilattices which are…

Interesting. Do you have any way of checking that the change has actually propagated through the system before starting to act on it? Is the system consistent at all times?

If I understand correctly, the client can connect to any instance and its request will get routed appropriately. Let's assume that you take a master offline and promote one of the replicas to be a new master. Won't that lead to a window in which (from the point of view of different instances) there are two masters at the same time and some writes are sent to the wrong instance?

EDIT:

One solution for such things is to use something like Zookeeper (or some other system whose documentation mentions "Paxos" ;)). Have you considered that? How does what you are doing compare with that?

Re: RethinkDB: An open-source distributed database built with love over three years

#118

Post by lacker »

jbellis wrote: post #5

I'll ask the obvious question not in the FAQ: How is this different from MongoDB?

One apparent difference between RethinkDB and MongoDB is that in RethinkDB, you can only index on the primary key. I imagine secondary indexes will be coming along soon.

Re: RethinkDB: An open-source distributed database built with love over three years

#119

Post by VeejayRampay »

tjic wrote: post #39

What the heck does "built with love" even mean? Is this just a hipster marketing term to tell us that it's small and cute and made by people who play ukuleles and ride unicycles in their spare time, and not by evil corporate people who commute to work and have mortgages? I find a lot of advertising eyeroll inducing, and the current trend of more-hipster-than-thou posturing is right at the top.

You look like an angry person man, chill the fuck out.

Rule of thumb is if you build something this nice and with that order of magnitude in complexity you can put My Little Poney stickers on your homepage and still get respect. Who cares about the "attitude" and the "language" for Christ's sake, they BUILT stuff with their own hands and are offering it to the world, they can do whatever they damn please.

Re: RethinkDB: An open-source distributed database built with love over three years

#120

Post by lttlrck »

crazygringo wrote: post #75
perfunctory wrote: post #61

Please stop using json as a data model. I have no idea how to represent dates, or timestamps, or colors, or any other unsupported data type.

I have nothing to do with RethinkDB, but what are you talking about? Just represent them as strings. (What databases support colors as native types, anyways?) If you format dates YYYY-MM-DD, then you can do string comparisons for ranges. And JSON has the huge advantage of supporting hierarchical data -- arrays with objects inside, etc. It seems a like a huge step forward.

Or unix epoch if you want an easy life when it comes to querying.
Post reply on HN