Live data from Hacker News

RethinkDB: An open-source distributed database built with love over three years

rethinkdb.com

181–190 of 247 posts

Re: RethinkDB: An open-source distributed database built with love over three years

#181
post #161
post #159

Earlier quoted context omitted.

Seems to me that most of us who have used MacPorts have moved to Homebrew or that could just be the bubble I'm living in. Is there anyone still who still uses MacPorts who could chime in and say why they never made the switch?

Come on, Homebrew doesn't even have gcc. I am not a Mac user, but a designer using MacBook joined our team last week, and we struggled for half a day with Homebrew. The next day, we installed MacPorts instead, and with just: $ sudo port install python27 py27-virtualenv gcc46 we were able to proceed and get the whole stack up and running. Not to mention everything from MacPorts is installed nicely under /opt/local. Ma…

Interesting. This has not been my experience at all.

I had to fight for days to get MacPorts to install anything properly. It gives me flashbacks to the horrors from 3-4 years ago of compiling open source software on Linux.

Homebrew has been fuzzy kittens in comparison.

Re: RethinkDB: An open-source distributed database built with love over three years

#182
post #152

Congratulate on releasing. Well done! A few questions: 1. Will secondary indices be ever supported? Range scan with a different order than the primary key is very welcomed. E.g. date range query. 2. Do you support conditional update? Or any kind of optimistic locking or versioning to coordinate concurrent updates from different clients? 3. Related to 2. How can loosely-sequential Id be generated using a table? 4. Wil…

1. Yes. It's a matter of doing this right, which will take some time.

2. Yes. There is no special command, you just combine update and branch (http://www.rethinkdb.com/api/#py:control_structures-branch) Here's an example in Python:

  r.table('foo').get(5).update({ 'bar': r.branch(r['baz'] == 0, 1, 2)})
This will set attribute bar to 1 if baz is 0, or to two 2 otherwise. Everything is atomic on that document.

3. Currently the server doesn't support a sequential (or even loosely sequential) id autogeneration. You'd have to do that on the clients, but using a timestamp for example.

4. I don't know yet how to do this really efficiently. It's relatively easy to do on a single shard, but cross-shard boundaries make this really hard.

5. Any client can connect to any server. The server will then parse and route the query. There is no central server, everything is peer-to-peer. The client library doesn't know about multiple servers now, so responsibility is on the user to hit a random server. Alternatively you can run "rethinkdb proxy" on localhost and connect the client to that. The proxy will then route queries to proper nodes in the cluster.

6. In the web UI, if you click on the table and reshard, everything will be rebalanced. You don't even have to add or remove shards, it'll just rebalance data for the number of shards you have. The UI has a bar graph with shard distribution, so you can see how balanced things are.

7. Currently there is no authentication support - we expect users to use proper firewall/ssh tunneling precautions.

8. Yes, that's how queries get routed. Currently this isn't very smart, but it will get much better over time. If something breaks for you performance-wise, just reach out and we'll fix it.

9. No, not yet. If you run eq_join on a small subset of the data (99% of OLTP workloads) it will be very fast. Other joins work ok, but there's A LOT of room for optimization.

Phew!

Re: RethinkDB: An open-source distributed database built with love over three years

#183
post #148

Earlier quoted context omitted.

Does the function have to be referentially transparent?

No. E.g., you could write r.table('foo').update(lambda row: row.merge({'bar': row['bar'] + 1 })). A shortcut for this is r.table('foo').update({'bar': r['bar'] + 1 }). Neither is referentially transparent, and both work.

This is beyond awesome, and thanks for the follow-ups!

Re: RethinkDB: An open-source distributed database built with love over three years

#184
post #161
post #159

Earlier quoted context omitted.

Seems to me that most of us who have used MacPorts have moved to Homebrew or that could just be the bubble I'm living in. Is there anyone still who still uses MacPorts who could chime in and say why they never made the switch?

Come on, Homebrew doesn't even have gcc. I am not a Mac user, but a designer using MacBook joined our team last week, and we struggled for half a day with Homebrew. The next day, we installed MacPorts instead, and with just: $ sudo port install python27 py27-virtualenv gcc46 we were able to proceed and get the whole stack up and running. Not to mention everything from MacPorts is installed nicely under /opt/local. Ma…

I'm a relatively new Mac user and never used MacPorts. When I search brew I get these results:

$ brew search gcc

apple-gcc42 gcc

homebrew/versions/gcc45 homebrew/versions/llvm-gcc28

I'm a bit confused since I thought that these are gcc. Can someone tell me what those results mean? :S

Re: RethinkDB: An open-source distributed database built with love over three years

#185

Is schemaless a win over an object schema like a JSON schema (or a Protocol Buffer .proto file)? Schemaless is clearly a convenience win over SQL because SQL's way of modeling nested/repeated data doesn't map as easily onto programming languages. But for all the people who are using JSON-based databases these days, I'm curious how many of them couldn't easily write a JSON schema or a .proto file that describes their…

this. I don't like SQL columns. They make life hard. But I'm spending time learning TypeScript specifically so I can add some types/schemas to my JavaScript. That doesn't mean I want to deal with the implementation detail of columns, but I definitely wouldn't mind some type safety.

I hear a lot of talk about how hard it is to maintain sql schemas and columns. I've used mongodb on projects previously and while it was interesting I didn't find that the lack of schema made life any easier. Writing a migration in rails is so easy, I just can't understand how managing a schema really makes life more difficult. What is it about managing a schema that makes people so eager to jump to schema-less?

Re: RethinkDB: An open-source distributed database built with love over three years

#186
post #152

Congratulate on releasing. Well done! A few questions: 1. Will secondary indices be ever supported? Range scan with a different order than the primary key is very welcomed. E.g. date range query. 2. Do you support conditional update? Or any kind of optimistic locking or versioning to coordinate concurrent updates from different clients? 3. Related to 2. How can loosely-sequential Id be generated using a table? 4. Wil…

1. Yes. It's a matter of doing this right, which will take some time. 2. Yes. There is no special command, you just combine update and branch ( http://www.rethinkdb.com/api/#py:control_structures-branch ) Here's an example in Python: r.table('foo').get(5).update({ 'bar': r.branch(r['baz'] == 0, 1, 2)}) This will set attribute bar to 1 if baz is 0, or to two 2 otherwise. Everything is atomic on that document. 3. Curre…

Thanks for your and jdoliner's detail answers! Hope I didn't ask too many questions. :) I'll respond to both here.

For 2 and 3, I think I didn't make it clear. Let me clarify. A common db problem with multiple clients is dealing with concurrent update on the same piece of data. E.g both client1 and client2 read D as D=15 at the same time. Client1 adds 1 to D as 16 and saves it. Then client2 adds 1 to D as 16 and save it as 16, which is wrong. It should be 17.

Conditional update is one feature db usually provides to let clients deal with this problem, i.e. the update would only go through if certain condition is met otherwise abort. Update D=16 if D==15. Client1 would succeed while client2 would fail, where it can retry the whole read-increment-update cycle again with the new read value.

The litmus test to see if a db system can handle this problem is to try to implement a sequential Id generation feature run by multiple clients at the same time.

For 8, if the query is parsed into a query execution plan, you can ship the plan to all equivalent replicas to ask them to estimate the execution cost based on their current load. After they reply, pick the lowest cost one and send the execute command. Even a simple approach of asking for machine load of all replicas and picking the lowest one could have adaptive utilization of all the servers.

For 9, Bloomer Filter is a relative simple technique that can dramatically reduce the amount of data to ship across peers to do join. You basically filter out the vast majority of the non-matching data before shipping.

It's a good start. Good luck going forward!

Re: RethinkDB: An open-source distributed database built with love over three years

#187

Please stop using json as a data model. I have no idea how to represent dates, or timestamps, or colors, or any other unsupported data type.

json definitely has certain advantages. e.g. arbitrarily nested data structures. However. json is not extensible by design. There is NO standard way to define new custom data types in json.

Re: RethinkDB: An open-source distributed database built with love over three years

#188
post #67

Nice work! It seems that you are well aware of the tradeoffs that you are taking and communicating it openly in your documentation (and your choices seem to be very reasonable). I really like the tone of your communication – it seems essentially BS/koolaid free. 1. How much data can you put in one instance before seeing performance degradation? I know that you still working on good benchmarks – but do you have any ba…

Hi, here to answer question number 4. Short answer: Our configuration data is most similar to git. Any machine can be used as an administrative node via the WebUI or the CLI. It will make changes to the metadata which then get pushed to the other nodes. If 2 nodes make conflicting changes you get a conflict which the system will help you to merge. Long Answer Cluster configuration is stored in semilattices which are…

Now that you mention it, it would be very nice to have a database suited for configuration that behaved like git in that branching, restoring old states, reverting selected commits was built in, while at the same time supporting ACID features and replication?

Does anyone know if something like that exist?

Re: RethinkDB: An open-source distributed database built with love over three years

#189
post #161

Earlier quoted context omitted.

Come on, Homebrew doesn't even have gcc. I am not a Mac user, but a designer using MacBook joined our team last week, and we struggled for half a day with Homebrew. The next day, we installed MacPorts instead, and with just: $ sudo port install python27 py27-virtualenv gcc46 we were able to proceed and get the whole stack up and running. Not to mention everything from MacPorts is installed nicely under /opt/local. Ma…

I'm a relatively new Mac user and never used MacPorts. When I search brew I get these results: $ brew search gcc apple-gcc42 gcc homebrew/versions/gcc45 homebrew/versions/llvm-gcc28 I'm a bit confused since I thought that these are gcc. Can someone tell me what those results mean? :S

This is the way to go:

    $ brew tap homebrew/dupes
    $ brew install gcc --enable-all-languages
To use your new gcc-4.7.2 when installing new packages just add '--use-gcc' at the end of the command.

Re: RethinkDB: An open-source distributed database built with love over three years

#190
post #161

Earlier quoted context omitted.

Come on, Homebrew doesn't even have gcc. I am not a Mac user, but a designer using MacBook joined our team last week, and we struggled for half a day with Homebrew. The next day, we installed MacPorts instead, and with just: $ sudo port install python27 py27-virtualenv gcc46 we were able to proceed and get the whole stack up and running. Not to mention everything from MacPorts is installed nicely under /opt/local. Ma…

clang is better than gcc, IMO

clang either does not build or defectively builds certain things on OS X, for instance Ruby 1.9.3. I had to acquire vanilla GCC for this reason the other day, and was relieved to find it in HomeBrew.
Post reply on HN