Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bedrock – Rock-solid distributed data

bedrockdb.com

Post reply on HN
Print view View original

51–60 of 110 posts

Re: Bedrock – Rock-solid distributed data

#51

Post by quinthar »

Animats wrote: post #50

So where are the other copies? The documentation tells you how to set up Bedrock locally, but to get redundancy, you need multiple copies talking. How do you set that up? And how does security work?

Here's a quick overview of how to configure in multiple zones: http://bedrockdb.com/multizone.html

Security isn't provided by Bedrock (or really, any database, at least not well). Rather, with Bedrock security is the responsibility of the application layer, but built into the database via stored procedures.

Re: Bedrock – Rock-solid distributed data

#52

Post by Jweb_Guru »

quinthar wrote: post #49
Jweb_Guru wrote: post #36

Earlier quoted context omitted.

The fundamental issue you're going to run into is that unless you have "genuine partial replication" where only certain shards have a key, or restrict your queries to key/value ones where the key can be determined automatically and used to route to a per-shard master, you can't detect conflicts committed on different nodes without executing half a round trip, which is going to be bottlenecked by the slowest node in y…

Thanks for all these links, I have some reading to do! Also, to clarify one point, we're not doing multi- master writes/replication -- just multi- threaded writes/replication. Incidentally, the latest plan is to use http://sqlite.org/sessionintro.html to do the following: 1) Spin up multiple write threads 2) Every write thread opens its own database handle 3) Every write thread creates a new "session" object before e…

Oh okay. That sounds a lot like OCC in terms of the tradeoffs involved. If you're looking for implementation hints, you might want to check out Silo (http://db.csail.mit.edu/pubs/silo.pdf). It's really hard to beat for transactions that are mostly nonconflicting (if transactions are mostly conflicting it is hard to exploit multiple writers unless you are very clever about how you go about it; I can dredge up a few links if you're interested). My overall understanding is that the usual rule is that unless you have highly partitionable workloads, single writer almost always wins, but I think your current strategy is also going to suffer from a lot of contention even in cases with no conflicts: the write batch is going to be constantly bouncing between cores and require lots of locking, which could easily substantially degrade performance over what it currently is (though obviously, benchmark :P).

From a correctness perspective: SQLITE_CONSTRAINT will only detect write conflicts, not read conflicts, so you are losing serializability (instead you have snapshot isolation, which is what SQLite gives you by default with multiple writers). SI allows a variety of subtle anomalies that can screw you over; see https://wiki.postgresql.org/wiki/SSI and https://wiki.postgresql.org/wiki/Serializable. If you are currently using only a single-writer thread, you will never experience these issues, AFAIK, because you cannot form a "dangerous structure" (you can never have two R-W dependencies that conflict with each other because you can only have one R-W transaction at a time), but I could be wrong about that; however, if I'm not, this could lead to you dealing with some really bad bugs. Note that these kinds of anomalies have a tendency to pop up when you're using stored procedures to perform validation!

Re: Bedrock – Rock-solid distributed data

#53

Post by quinthar »

bithavoc wrote: post #46
kbaker wrote: post #37

OK, how is this different from Rqlite [1], except in C++ instead of Go, Paxos instead of Raft? > rqlite is a distributed relational database, which uses SQLite as its storage engine. > rqlite gives you the functionality of a rock solid, fault-tolerant, replicated relational database, but with very easy installation, deployment, and operation. Hmmm.... Also, is Bedrock DB only 30 days old? (since the 'first commit' me…

Also with identical approach there is http://www.actordb.com

Oh, neat! That one seems to replace SQLite's storage engine with a custom one. But yes, also similar. Cool! Again, what is the largest real-world user? We've found a lot of the theories tend to break down when they are subjected to reality -- Bedrock's advantage is it's been hardened in the crucible of live production traffic for years.

Re: Bedrock – Rock-solid distributed data

#54

Post by quinthar »

quinthar wrote: post #42
chucky_z wrote: post #40

Earlier quoted context omitted.

Automatic failover -- https://dev.mysql.com/doc/mysql-utilities/1.5/en/mysqlfailov... Guides to doing auto failover and recovery: https://dev.mysql.com/doc/mysql-utilities/1.5/en/utils-task-... https://dev.mysql.com/doc/mysql-utilities/1.5/en/utils-task-...

Cool, thanks! It looks like that came out in 2012? Neat, I'll take a look!

Incidentally, if anybody has experience with this I'd love to know:

1) What happens if "mysqlfailover" itself dies? For example, if I have 6 servers split equally between 2 datacenters, and the datacenter running mysqlfailover loses power -- how does the other datacenter get reconfigured?

2) If you run two copies of mysqlfailover (one in each datacenter), how does it solve the "split brain" problem? If you have 6 servers split equally between two datacenters, and a clean network severance down the middle, what prevents both sides from configuring and operating a master? (This is the nightmare scenario.)

3) If mysqlfailover dies, it sounds like it will prevent any other copy from running without manual intervention: "At startup, the console will attempt to register itself with the master. If another console is already registered, and the failover mode is auto or elect, the console will be blocked from running failover. When a console quits, it unregisters itself from the master. If this process is broken, the user may override the registration check by using the --force option."

Overall, this sounds like a great improvement over what existed before (eg, nothing), but still a pretty brittle, manual, and (in the case of split-brain) very dangerous approach. Have you had good experience with it in practice?

Re: Bedrock – Rock-solid distributed data

#55

Post by quinthar »

Jweb_Guru wrote: post #52
quinthar wrote: post #49

Earlier quoted context omitted.

Thanks for all these links, I have some reading to do! Also, to clarify one point, we're not doing multi- master writes/replication -- just multi- threaded writes/replication. Incidentally, the latest plan is to use http://sqlite.org/sessionintro.html to do the following: 1) Spin up multiple write threads 2) Every write thread opens its own database handle 3) Every write thread creates a new "session" object before e…

Oh okay. That sounds a lot like OCC in terms of the tradeoffs involved. If you're looking for implementation hints, you might want to check out Silo ( http://db.csail.mit.edu/pubs/silo.pdf ). It's really hard to beat for transactions that are mostly nonconflicting (if transactions are mostly conflicting it is hard to exploit multiple writers unless you are very clever about how you go about it; I can dredge up a few…

Hm... thanks for the head's up. I thought that SQLITE_CONSTRAINT would detect both read and write conflicts, but I'll dig deeper there.

As for how much contention there will be, I'm not sure: that's one reason we haven't taken the plunge yet as we sorta want to determine this first. However, given that we have a huge number of users modifying unshared data (eg, our largest group still makes up a tiny fraction of the userbase), I suspect conflicts will be relatively few.

And to be clear: even a little write concurrency goes a long way. If we can even do two writes simultaneously, that doubles our write performance. But I personally suspect the average batch size of non-conflicting transactions will be in the dozens or even hundreds -- meaning I suspect we will achieve so much replication throughput that something else becomes the bottleneck before replication is saturated.

Only one way to find out however!

Re: Bedrock – Rock-solid distributed data

#56

Post by drkp »

Jweb_Guru wrote: post #52
quinthar wrote: post #49

Earlier quoted context omitted.

Thanks for all these links, I have some reading to do! Also, to clarify one point, we're not doing multi- master writes/replication -- just multi- threaded writes/replication. Incidentally, the latest plan is to use http://sqlite.org/sessionintro.html to do the following: 1) Spin up multiple write threads 2) Every write thread opens its own database handle 3) Every write thread creates a new "session" object before e…

Oh okay. That sounds a lot like OCC in terms of the tradeoffs involved. If you're looking for implementation hints, you might want to check out Silo ( http://db.csail.mit.edu/pubs/silo.pdf ). It's really hard to beat for transactions that are mostly nonconflicting (if transactions are mostly conflicting it is hard to exploit multiple writers unless you are very clever about how you go about it; I can dredge up a few…

You're correct that SI with a single writer will always be serializable.

(I was one of the authors of SSI in Postgres)

Re: Bedrock – Rock-solid distributed data

#58

Post by Jweb_Guru »

quinthar wrote: post #55
Jweb_Guru wrote: post #52

Earlier quoted context omitted.

Oh okay. That sounds a lot like OCC in terms of the tradeoffs involved. If you're looking for implementation hints, you might want to check out Silo ( http://db.csail.mit.edu/pubs/silo.pdf ). It's really hard to beat for transactions that are mostly nonconflicting (if transactions are mostly conflicting it is hard to exploit multiple writers unless you are very clever about how you go about it; I can dredge up a few…

Hm... thanks for the head's up. I thought that SQLITE_CONSTRAINT would detect both read and write conflicts, but I'll dig deeper there. As for how much contention there will be, I'm not sure: that's one reason we haven't taken the plunge yet as we sorta want to determine this first. However, given that we have a huge number of users modifying unshared data (eg, our largest group still makes up a tiny fraction of the…

Thinking about it, your algorithm also does not sound crash-resilient. Specifically, unless you're doing round-trip confirmations, you can easily have different replicas with different (possibly overlapping, possibly disjoint, in weird patterns) sets of changesets applied within a batch. If you have asynchronous replication at all you probably already have to deal with any issues arising from "gaps" in replication on some of the replicas for a single version, but if you don't, be aware that it can be very tricky to deal with properly on recovery. Specifically, if the master applies different transactions (within a batch) to different replicas, then goes down, and you elect a replica that only saw some of the transactions, and there can't be gaps, it's relatively easy to identify which replica is most up-to-date and make that one the leader. But if there can be gaps, there's the possibility that different nodes have different transactions applied, so you'll have to make sure any gaps are filled in before you elect a new leader.

Still, with a total order, it's not that bad, because it's easy to identify gaps; with your algorithm, however, there's no total order between changesets within or across batches (at least, none that I can see in your algorithm), so you won't have any way of knowing whether there are gaps (at least, not without doing something silly like broadcasting every changeset)! Adding a total order seems like the easiest way to resolve that, but like I said even that case can be tricky. Also, your algorithm doesn't specify that replicas which receive changesets with batch ids with gaps in them wait to apply them until the gap is filled in (which means they have to know when the previous batch is done), or alternately that before a batch with a new id can be sent from the master all replicas must have synchronously applied all changes within the old batch. I think something like that is a requirement here in order for there not to be potential conflicts on your replicas during replication even if the master doesn't crash.

IMO, doing exactly your algorithm but delaying replication or transaction confirmation until a specified amount of time has passed (say 10 ms), then synchronously sending the entire changeset and incrementing the batch id (aka group commit) is a much better idea. You exchange marginally worse latency (and more bursty network usage) for a far less complex update scenario (yes, you're back to having a total ordering on writes at replicas you need to follow, but you can include multiple changesets in the batch). You can also have your master monitor number of writes and only turn on group commit if there's unusually high write volume.

(Also, belatedly reading one of your other comments more carefully, I'm fairly confident commitCount will also not work correctly if changesets can be applied out of order on different replicas, even if there was no data loss, since the same commitCount could include different sets of changesets at any one time).

Re: Bedrock – Rock-solid distributed data

#59

Post by Jweb_Guru »

drkp wrote: post #56
Jweb_Guru wrote: post #52

Earlier quoted context omitted.

Oh okay. That sounds a lot like OCC in terms of the tradeoffs involved. If you're looking for implementation hints, you might want to check out Silo ( http://db.csail.mit.edu/pubs/silo.pdf ). It's really hard to beat for transactions that are mostly nonconflicting (if transactions are mostly conflicting it is hard to exploit multiple writers unless you are very clever about how you go about it; I can dredge up a few…

You're correct that SI with a single writer will always be serializable. (I was one of the authors of SSI in Postgres)

And Tapir too, I see... and Arrakis? And speculative Paxos? Keep up the awesome work, I always learn stuff from your papers :) And thanks for confirming, I hadn't actually thought about SI in the single-writer case until that comment.

Re: Bedrock – Rock-solid distributed data

#60

Post by HammadB »

quinthar wrote: post #16
thesmallestcat wrote: post #14

Earlier quoted context omitted.

And then laugh maniacally when the partition heals. What's a "session"? Can you explain commitCount more, and how a client is supposed to use it intelligently without deadlocking. Also there's a lot of master/slave talk and I'm wondering how leaders are elected during a partition, and what recovery looks like.

Heh, there's a lot of detail to be captured I agree. But in short: - All nodes connect to all other nodes - Each node has a priority; a Paxos algorithm is used to identify the highest priority node, which "stands up" to be the master - All nodes respond to read queries using the local database. - So long as you always talk to the same node, there are no consistency issues: you are guaranteed that each request will an…

Are you doing Paxos across a WAN? Isn't this slow?
Post reply on HN