Discuss in real-time here: https://gitter.im/Expensify-Bedrock/Lobby
Bedrock – Rock-solid distributed data
11–20 of 110 posts
Re: Bedrock – Rock-solid distributed data
#12Rock-solid? Look. Just about anyone can create some "database" and declare it "rock-solid". Frankly, a simple key-value store is built in most languages. Do you have more than that? Do you have benchmarks? You know, competing against the other providers?
It must be Bedrock Linux' standard DB.
Re: Bedrock – Rock-solid distributed data
#13Earlier quoted context omitted.
Heh, yep, we definitely had that sent to us. The point of Bedrock is to resolve those points and make SQLite usable in the enterprise. As for the CAP theorem, we compromise on "C". Each slave responds based on its current state, which might be a different state than other slaves or the master. However, every response returns the "commitCount" of the database at the time the response was generated. Then when you submi…
So you're ensuring consistency by blocking on all queries? And it's up to the client to enforce this to any degree? And if there's one master and a global clock, sounds like you threw out the "P" too. Frankly this thing sounds simplistic and unusable, and it's sorta offensive to hear SQLite called unusable in the enterprise, whatever that means.
Consistency is only an issue if you switch database servers mid session. Just "stick" to a node by issuing all subsequent requests to the same server and there are no consistency issues.
Re: Bedrock – Rock-solid distributed data
#14Earlier quoted context omitted.
So you're ensuring consistency by blocking on all queries? And it's up to the client to enforce this to any degree? And if there's one master and a global clock, sounds like you threw out the "P" too. Frankly this thing sounds simplistic and unusable, and it's sorta offensive to hear SQLite called unusable in the enterprise, whatever that means.
Consistency is only an issue if you switch database servers mid session. Just "stick" to a node by issuing all subsequent requests to the same server and there are no consistency issues.
And then laugh maniacally when the partition heals. What's a "session"? Can you explain commitCount more, and how a client is supposed to use it intelligently without deadlocking. Also there's a lot of master/slave talk and I'm wondering how leaders are elected during a partition, and what recovery looks like.
Re: Bedrock – Rock-solid distributed data
#15I don't want to be a jerk but... just for the sake of common sense: - Faster? Provide a benchmark. Faster is a relative thing, faster than what? faster under which conditions? - More reliable... than what? under which conditions? how do you know it is more reliable? - More powerful? I thought SQLite is more constrained than other SQL databases. e.g: No stored procedures. Powerful? how? compared to exactly what? Nulli…
Thanks for all that Latin! Yes, it's been open sourced for under 24 hours. Those are all good suggestions.
Una lingua numquam satis est. Gratias tibi.
Good luck with your project.
Re: Bedrock – Rock-solid distributed data
#16Earlier quoted context omitted.
Consistency is only an issue if you switch database servers mid session. Just "stick" to a node by issuing all subsequent requests to the same server and there are no consistency issues.
And then laugh maniacally when the partition heals. What's a "session"? Can you explain commitCount more, and how a client is supposed to use it intelligently without deadlocking. Also there's a lot of master/slave talk and I'm wondering how leaders are elected during a partition, and what recovery looks like.
Heh, there's a lot of detail to be captured I agree. But in short:
- All nodes connect to all other nodes
- Each node has a priority; a Paxos algorithm is used to identify the highest priority node, which "stands up" to be the master
- All nodes respond to read queries using the local database.
- So long as you always talk to the same node, there are no consistency issues: you are guaranteed that each request will answered with a database at least as fresh as the last.
- However, if you switch databases (eg, a node goes down and you are forced to go to a different one that might not be as fresh), it will wait until it is as fresh as the node you were using
- Write requests are escalated to the master, which processes them with a two-phase commit distributed transaction
- By default, the master waits for a "quorum" of the cluster to approve the transaction before committing
- However, this limits write capacity to 1/median(rtt) of the slaves
- Accordingly, we have a "selective synchronization" feature where you can optionally designate some transactions as requiring "less consistency". For example, you can specify that the master should commit when at least one other node approves.
- For the highest performance, you can designate a transaction as "asynchronous" and the master will commit immediately
- For example, when someone is reimbursed, we absolutely want to get quorum approval of the transaction: we can't risk losing that if the master crashes. But if someone just adds a report comment, we can risk losing it upon master crash.
- When the master does go down (either gracefully or not), the next highest priority node steps up seamlessly
- Any command escalated to the master but not processed will be re-escalated to the new master
- When a higher priority node comes up, it synchronizes to get up to date with everything it missed, and then becomes master
Anyway, lots of details here and I agree they haven't been well captured in the site yet. Coming soon!
Re: Bedrock – Rock-solid distributed data
#17Here's Expensify's CEO (David Barrett) posting on the p2p-hackers mailing list about this. It's from April 2016.
http://p2p-hackers.709552.n3.nabble.com/p2p-hackers-Advice-o...
Re: Bedrock – Rock-solid distributed data
#18Rock-solid? Look. Just about anyone can create some "database" and declare it "rock-solid". Frankly, a simple key-value store is built in most languages. Do you have more than that? Do you have benchmarks? You know, competing against the other providers?
Agreed, benchmarks would be helpful to back up these claims. Good suggestion!
I think the argument in favor of it being more reliable is an easier one, however, as Bedrock is specifically designed (and used, at scale, for 8 years) to provide highly available redundancy in the form of WAN-replication and automatic failover.
Regardless, I think you'll agree that "rock-solid" is not a precise technical term, and I hope you'll allow a bit of play on words. Whether or not it's literally as solid as a rock is unclear, but it is definitely very reliable -- and more reliable than most.
Re: Bedrock – Rock-solid distributed data
#19Here's Expensify's CEO (David Barrett) posting on the p2p-hackers mailing list about this. It's from April 2016. http://p2p-hackers.709552.n3.nabble.com/p2p-hackers-Advice-o...
Yes, we've made a lot of good progress on this front, though nothing ready to demo. Multi-threaded replication is really exciting stuff. As it stands, our "selective sync" capability already gives a lot of headroom for write capacity, but multi-threaded replication will raise that ceiling even higher.
Re: Bedrock – Rock-solid distributed data
#20I don't want to be a jerk but... just for the sake of common sense: - Faster? Provide a benchmark. Faster is a relative thing, faster than what? faster under which conditions? - More reliable... than what? under which conditions? how do you know it is more reliable? - More powerful? I thought SQLite is more constrained than other SQL databases. e.g: No stored procedures. Powerful? how? compared to exactly what? Nulli…
Incidentally, sqlite doesn't provide stored procedures, but Bedrock does (using a C++ plugin capability). Otherwise, SQLite is actually very feature rich; a subset of features are here: http://sqlite.org/fullsql.html