Earlier quoted context omitted.
Heh, there's a lot of detail to be captured I agree. But in short: - All nodes connect to all other nodes - Each node has a priority; a Paxos algorithm is used to identify the highest priority node, which "stands up" to be the master - All nodes respond to read queries using the local database. - So long as you always talk to the same node, there are no consistency issues: you are guaranteed that each request will an…
Are you doing Paxos across a WAN? Isn't this slow?
Bedrock – Rock-solid distributed data
61–70 of 110 posts
Re: Bedrock – Rock-solid distributed data
#62> Bedrock is designed from the ground up to operate in a multi-datacenter environment with rigorous use of stored procedures – the new standard for modern application design. stored procedures—new standard for modern application design?
Your webserver is the first thing to be hacked by an attacker because it's the server that sits on the internet. If all your security logic is built on your webserver, an attacker can easily bypass it all. However, if your security logic is built into your database, then it dramatically limits the damage a hacker can do from the webserver.
In particular, I would recommend you create an "Authenticate" stored procedure that accepts a username and password, returning an "authToken". Then make all your other functions into similar stored procedures, each of which accepts and verifies an "authToken" before returning the results.
(And in the case of Bedrock, you might also disable the Bedrock::DB plugin entirely to prevent direct access to your database from the webserver, forcing everybody to go through your stored procedures.)
This design means attackers who root your webserver can't randomly access your database without knowing the username/password (or valid authToken) -- all of which is neatly and securely contained inside the database itself.
To be clear, if you do keep your SQL on your webserver (as most applications honestly do, though I believe it's a mistake), then the Bedrock::DB plugin is perfect for you. But I think Bedrock::DB (or really, any direct access to a database outside of stored procedures) should be largely viewed as a programming and maintenance convenience, at the expense of security.
Re: Bedrock – Rock-solid distributed data
#63Earlier quoted context omitted.
Hm... thanks for the head's up. I thought that SQLITE_CONSTRAINT would detect both read and write conflicts, but I'll dig deeper there. As for how much contention there will be, I'm not sure: that's one reason we haven't taken the plunge yet as we sorta want to determine this first. However, given that we have a huge number of users modifying unshared data (eg, our largest group still makes up a tiny fraction of the…
Thinking about it, your algorithm also does not sound crash-resilient. Specifically, unless you're doing round-trip confirmations, you can easily have different replicas with different (possibly overlapping, possibly disjoint, in weird patterns) sets of changesets applied within a batch. If you have asynchronous replication at all you probably already have to deal with any issues arising from "gaps" in replication on…
Now I do agree that it's tricky to avoid "gaps" in the failure cases. However, every replica keeps a record of the past several million transactions (we aim for 3 days), and every transaction is assigned a unique ID. When a replica starts up, it "synchronizes" down every missing transaction it has, and at this point would "repair" any gaps it somehow obtained when it went down.
Admittedly, the exact details of that part are TBD, but it doesn't strike me as an unresolvable problem on the surface.
Re: Bedrock – Rock-solid distributed data
#64Earlier quoted context omitted.
Thinking about it, your algorithm also does not sound crash-resilient. Specifically, unless you're doing round-trip confirmations, you can easily have different replicas with different (possibly overlapping, possibly disjoint, in weird patterns) sets of changesets applied within a batch. If you have asynchronous replication at all you probably already have to deal with any issues arising from "gaps" in replication on…
To be clear, my sense is that every replica would receive every transaction, and would be free to commit the transactions inside each batch in any order. Now I do agree that it's tricky to avoid "gaps" in the failure cases. However, every replica keeps a record of the past several million transactions (we aim for 3 days), and every transaction is assigned a unique ID. When a replica starts up, it "synchronizes" down…
(1) In the event of a crash, how does it know which transactions it's missing without a total order on the write transactions?
(2) In the non-failure case, how do you know the transactions in one batch are ordered before the transactions in a subsequent batch (which requires the replica to identify any "gaps" within the previous batch so it can wait for them to come in before continuing on)?
I don't think any of this is unresolvable (just adding a total order would go a long way). I do think it's very tricky to get right, with lots of edge cases, and that if you can't efficiently identify missing transactions it potentially makes recovery unusably slow.
In any case, since I believe you are losing serializability with multiple writers, you should probably weigh that against any performance gains you get from multiple writers (I think even the group commit variant suffers from this problem).
Re: Bedrock – Rock-solid distributed data
#65* Jobs - Provides a simple job queue.
* Cache - Provides a simple replicated cache.
...while these look like nice-to-have features which make life easy for developers, IMO having a database which is also a queue and is also a cache is a way to end up with a mess you can't scale out and is hard to reason about.
Have seen this happen with Redis - unless you're disciplined about how you use it and deploy it, having db, cache and queue all in one place is a honeypot for developers that turns into a future nightmare for operations.
Re: Bedrock – Rock-solid distributed data
#66Earlier quoted context omitted.
You're correct that SI with a single writer will always be serializable. (I was one of the authors of SSI in Postgres)
And Tapir too, I see... and Arrakis? And speculative Paxos? Keep up the awesome work, I always learn stuff from your papers :) And thanks for confirming, I hadn't actually thought about SI in the single-writer case until that comment.
Re: Bedrock – Rock-solid distributed data
#67Re: Bedrock – Rock-solid distributed data
#68Earlier quoted context omitted.
Cool, thanks! It looks like that came out in 2012? Neat, I'll take a look!
Incidentally, if anybody has experience with this I'd love to know: 1) What happens if "mysqlfailover" itself dies? For example, if I have 6 servers split equally between 2 datacenters, and the datacenter running mysqlfailover loses power -- how does the other datacenter get reconfigured? 2) If you run two copies of mysqlfailover (one in each datacenter), how does it solve the "split brain" problem? If you have 6 ser…
We are using it since 2014. First in line is a HAProxy which directs the traffic to one of the 3 nodes. If one of the nodes goes down HAProxy directs the traffic to the other two nodes.
This works quite well, as long as not all 3 nodes are down / lose network connection. After that you have to initialise the cluster again. One node is the new master and all later changes in the other databases are overwritten. Then we connect each node to the new cluster.
The biggest problems are write intense operations. We had more luck using two nodes as read nodes and using the third node as write node. When the data in two nodes got updated simultaneously the performance suffered heavily. After some smarter scheduling it works pretty well (as long as not all the nodes go down or the network connection between the nodes is lost). The slowest node can block the write operations for all the other nodes.
Re: Bedrock – Rock-solid distributed data
#69I don't quite get why SQLite is the right primitive to compose a large scale database system out of... If the design goal is write throughput, SQLite can be beat really easily by the client/server systems since each SQLite instance only supports one concurrent writer. I guess users of Bedrock could partition their datasets into different pieces that aren't often written in tandem, but why go through all the trouble i…
Thanks for asking! Re: "SQLite can be beat really easily by the client/server systems since each SQLite instance only supports one concurrent writer" -- That's not actually true, sqlite supports concurrent writers via their page-locking branch (and changesets allow for effectively row-level locks). But none of that matters, because single-threaded replication means any multi-threaded write capability is irrelevant. R…
I've been unable to find the branch on SQLite website. Do you have a link to the branch and changesets?
Re: Bedrock – Rock-solid distributed data
#70The idea's not fundamentally bad, but some of the rhetoric raises alarm bells. > built atop SQLite, the fastest, most reliable, and most widely distributed database in the world. Most widely distributed maybe, but far from the fastest and AFAIK others beat it on reliability as well. No disrespect to SQLite, it's far more feature-rich than other embedded-style DBs, but without a definition of "database" that excludes…
Re: SQLite's performance -- Can you point to anything that shows SQLite isn't the fastest for some class of query? In my experience (though admittedly I haven't formally benchmarked) it's at least as fast, and generally faster (due to the dramatically reduced overhead to query it over a traditional database server). Re: SQLite reliability. I'm confused how SQLite isn't more reliable than every other contender, merely…
I'm curious: how do you keep track of this number?