Live data from Hacker News

Bedrock – Rock-solid distributed data

bedrockdb.com

31–40 of 110 posts

Re: Bedrock – Rock-solid distributed data

#31

Earlier quoted context omitted.

Out of curiosity, what specifically do you find idiotic about the document you linked to?

I didn't call anything idiotic. But let's just say that grappling with the STL may be slightly preferable to this... thing. And there is some special irony referring to STL as an "esoteric abstraction" while offering your own vastly more esoteric abstractions in the same swoop. And the following paragraph speaks for itself, particularly in the context of developing a fucking database : > Computers are super fast thes…

Yes, I think it does speak for itself. Thanks!

Re: Bedrock – Rock-solid distributed data

#32
Great idea!!! SQLite is an extraordinary product. I'll be following Bedrock.

This could be, for databases, what node was for server-side programming. There are parallels, you're using a "client side single-writer (excellent) DB" and making it apt for the cloud with Paxos. Again, great idea!!!!

And it seems you're taking with a good humor some "strong" comments here. +1 sr. +1 to u.

Re: Bedrock – Rock-solid distributed data

#33

Earlier quoted context omitted.

Yes... but you said "More powerful", rather than just "Powerful". More implies a comparison with respect to something else... with what specifically?

Well, with respect to stored procedures, I think C++ is a far more powerful language than MySQL's SQL-based approach -- especially since Bedrock plugins can also encapsulate schema changes. I think Bedrock's replication is definitely more more powerful than MySQL's as well. So in general I think it's more powerful than MySQL. I'd love to hear your thoughts about where MySQL has the edge, however. Thanks!

Does it have transactions?

Re: Bedrock – Rock-solid distributed data

#34
post #32

Great idea!!! SQLite is an extraordinary product. I'll be following Bedrock. This could be, for databases, what node was for server-side programming. There are parallels, you're using a "client side single-writer (excellent) DB" and making it apt for the cloud with Paxos. Again, great idea!!!! And it seems you're taking with a good humor some "strong" comments here. +1 sr. +1 to u.

Thanks, strong comments welcome!

Re: Bedrock – Rock-solid distributed data

#36

Here's Expensify's CEO (David Barrett) posting on the p2p-hackers mailing list about this. It's from April 2016. http://p2p-hackers.709552.n3.nabble.com/p2p-hackers-Advice-o...

Yes, we've made a lot of good progress on this front, though nothing ready to demo. Multi-threaded replication is really exciting stuff. As it stands, our "selective sync" capability already gives a lot of headroom for write capacity, but multi-threaded replication will raise that ceiling even higher.

The fundamental issue you're going to run into is that unless you have "genuine partial replication" where only certain shards have a key, or restrict your queries to key/value ones where the key can be determined automatically and used to route to a per-shard master, you can't detect conflicts committed on different nodes without executing half a round trip, which is going to be bottlenecked by the slowest node in your "fast" quorum. That's why nobody uses Generalized Consensus in practice.

Batching before that will help if you already know that most of the transactions touching a particular key are going to the same server. So not a clean partition, but "most of the time". This is the idea behind, e.g., http://www.ssrg.ece.vt.edu/papers/peluso-M2PAXOS-TR.pdf. You try to make sure people usually contact the owner of a particular partition, and use a variety of techniques from the literature to make that well-optimized (ideally, clients actually know which partitions they "should" be accessing, too).

Another approach is to restrict your transactions to deterministic ones (with possible "scouting" transactions to do things like secondary index lookups where you can't statically analyze which partitions they will hit), in which case you can batch all transactions, send them to all participant nodes, and run them afterwards without two-phase commit. This is the approach taken by Calvin (which can exploit per-node parallelism during execution because it uses deterministic, deadlock-free locking). Additionally, read-only transactions scoped to a partition can execute locally without any distributed access, because thanks to determinism serializability is guaranteed. See http://cs-www.cs.yale.edu/homes/dna/papers/calvin-tods14.pdf (and yes, it really can do 500,000 distributed TPC-C transactions per second--by my estimation, more than three orders of magnitude higher than the average total traffic you get, as suggested by your "100x more read transactions than write" estimate. Though of course your current average is higher than your 8-year average, I doubt it is 1000x higher).

Yet another idea is to try to combine coordination required for replication with that required for the distributed transaction, as Tapir does: https://github.com/UWSysLab/tapir. It enjoys the highest linearizable read/write "general" (as in, doesn't have to be submitted in one batch or need static analysis) transaction throughput I've seen of any leaderless georeplicated system (aka it processes transactions "fairly").

Still another approach is to forego serializability for a very slightly weaker guarantee, extended update serializability, which eliminates nearly all interesting anomalies but can allow for dramatically better performance: http://www.ssrg.ece.vt.edu/papers/opodis14-alvin.pdf demonstrates that on read-mostly workloads you can do very well with that even in absurd deployments (e.g. the georeplicated 7 datacenter one in the paper). Elsewhere in the thread you were talking about how "observational" consistency is what's important, which would suggest you are probably already relying on the "strong session" assumption (in which case EUS is indistinguishable from serializability), so I encourage you to give that a look.

All four represent pretty interesting points on the design space. It's not clear to me whether RockSolid would be better than, say, SQLite over Calvin, for your use case (in particular, I suspect Expensify would not find Calvin's requirement that transactions be statically analyzable terribly onerous, especially since you already require all transactions to be executed as stored procedures).

I will say that I'm pleased that a site receiving a reasonable amount of traffic is using SQLite. People consistently underestimate its performance.

Re: Bedrock – Rock-solid distributed data

#37
OK, how is this different from Rqlite [1], except in C++ instead of Go, Paxos instead of Raft?

> rqlite is a distributed relational database, which uses SQLite as its storage engine.

> rqlite gives you the functionality of a rock solid, fault-tolerant, replicated relational database, but with very easy installation, deployment, and operation.

Hmmm....

Also, is Bedrock DB only 30 days old? (since the 'first commit' message on GitHub?) That's barely topsoil!

[1] https://github.com/rqlite/rqlite

Re: Bedrock – Rock-solid distributed data

#38
I think it is important to encourage people to build their own databases, so I want to commend you guys for doing it. But I do have some questions (disclosure: I am the author of https://github.com/amark/gun ):

- Is it correct to say that Bedrock is primarily a replication layer for SQLite? Most things your homepage highlight seem to be SQLite features, not Bedrock features.

- Elsewhere in this thread you mention you run PAXOS (what implementation are you using? Your own? Do you have a test we can run to verify correctness?) But then you mention you sacrifice the "C" in the CAP Theorem, which would go against any point of having PAXOS. What happens in a split-brain scenario, where you have lets say 6 peers, and there is a network partition straight down the middle?

- You also mention elsewhere in this thread that if you are talking to the same node (a sticky session) there is no consistency issues. Do you mean linearizable? Because consistency has to do with multi-machine data consistency. For instance, if a write happens on another peer, then the node you are talking to should return that write not a stale write.

- Is there anything other than just assuming it is run on SSD that make it faster than SQLite on SSD?

Thanks!

Re: Bedrock – Rock-solid distributed data

#39
post #35

How does the client get "redirected" to another node if the one you're talking to fails? It is transparent to the client?

Our PHP library has the concept of a primary and failover host. We recommend configuring a local Bedrock node as the "primary host", and then setting up a load-balanced pool of all nodes to be used as the "secondary host". This means in normal operation, every webserver talks directly its nearest node (bypassing the load balancer), but "fails over" to the load balancer if the primary is down.

Granted, a simpler configuration is to just have two load balancers -- both of which have all nodes -- and then use each as the primary/secondary. We did this for years and it was fine, but for just a tiny bit more configuration overhead you can bypass the load balancer for the primary and thus shave off those precious milliseconds from each request.

Re: Bedrock – Rock-solid distributed data

#40

I don't quite get why SQLite is the right primitive to compose a large scale database system out of... If the design goal is write throughput, SQLite can be beat really easily by the client/server systems since each SQLite instance only supports one concurrent writer. I guess users of Bedrock could partition their datasets into different pieces that aren't often written in tandem, but why go through all the trouble i…

Thanks for asking! Re: "SQLite can be beat really easily by the client/server systems since each SQLite instance only supports one concurrent writer" -- That's not actually true, sqlite supports concurrent writers via their page-locking branch (and changesets allow for effectively row-level locks). But none of that matters, because single-threaded replication means any multi-threaded write capability is irrelevant. R…

Automatic failover -- https://dev.mysql.com/doc/mysql-utilities/1.5/en/mysqlfailov...

Guides to doing auto failover and recovery:

https://dev.mysql.com/doc/mysql-utilities/1.5/en/utils-task-...

https://dev.mysql.com/doc/mysql-utilities/1.5/en/utils-task-...

Post reply on HN