Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bedrock – Rock-solid distributed data

bedrockdb.com

Post reply on HN
Print view View original

91–100 of 110 posts

Re: Bedrock – Rock-solid distributed data

#91

Post by quinthar »

toolslive wrote: post #73
quinthar wrote: post #43

Earlier quoted context omitted.

Re: SQLite -- Yes, Bedrock is just the replication layer, SQLite does all the SQL and storage. Re: Paxos -- It's our own implementation. Split brain is prevented by the master refusing to stand up unless a majority of configured (not just active) slaves approve its standup request. So in a 6 node deployment, 1 master and 5 slaves, 3 of the slaves would need to approve. This means one "half" would have 4 nodes, and th…

Is the set of servers that participate a paxos value too (ie determined using the consensus algorithm), or is it configured ? Can you grow a cluster without bringing it down ?

Unfortunately you can't add new nodes without reconfiguring the cluster, and currently that requires restarting each server (though so long as you don't do it all at once, server restarts are normal maintenance that causes no downtime to the end user). This could likely be added without too much effort, but also the real-world use case of this is uncertain.

The consensus algorithm is only used to elect a master -- once the master is identified, it coordinates all the distributed transactions. If the master dies, everyone who remains elects a new master and re-escalates all unprocessed write transactions to the new master.

Read transactions are processed locally by each node, so only writes need to be escalated.

Re: Bedrock – Rock-solid distributed data

#92

Post by ngrilly »

quinthar wrote: post #84
ngrilly wrote: post #70

Earlier quoted context omitted.

> 2158341079 writes I'm curious: how do you keep track of this number?

Every write transaction ever committed to the database is assigned a unique ID, and the most recent few million transactions are kept in a "journal" table. This is what allows nodes to figure out what happened while they were offline and re-execute the missing transactions. A side-effect of that is we know exactly how many write transactions have ever been committed.

That's a funny side-effect! Should I understand the transaction unique ID is an incrementing number?

Re: Bedrock – Rock-solid distributed data

#93

Post by ngrilly »

quinthar wrote: post #87
ngrilly wrote: post #77

Two questions: If I'm not mistaken, creating an index is a blocking operation in SQLite (other writers are blocked). How do you manage this in production? How do you create a stored procedure? Do you have to recompile the whole program? How do you deploy with zero downtime?

Great questions! Yes, indexing is actually a significant challenge at our scale, as it can take a long time to add and currently, it is a blocking operation. However, the sqlite team is amazing and has all sorts of write-concurrency tricks up their sleeve to allow for creating indexes in a parallel thread. We haven't used it yet, but we plan to. As for the stored procedures, yes they are written in C++, and thus depl…

> Yes, indexing is actually a significant challenge at our scale, as it can take a long time to add and currently, it is a blocking operation.

What do you do during index creation: Do you drop write requests on the floor? Do you store them in a persistent queue somewhere? Or do you remove a node from the cluster, create the index, re-add the node to the cluster, and repeat this on every other nodes?

I'm asking because at some point I thought of using SQLite, embedded in a process containing the business logic and the network layer, but I abandoned this idea specifically because of blocking operations like index creation, and decided to use PostgreSQL instead.

> the sqlite team is amazing and has all sorts of write-concurrency tricks up their sleeve to allow for creating indexes in a parallel thread

I'd be very interested in any link related to this new feature of SQLite.

> However, we have 6 of them (and honestly, any one of which has enough read capacity to satisfy about all our traffic) so a minute of downtime to upgrade for each independently isn't a problem.

So, basically, you're doing a rolling upgrade? Have you evaluated compiling your stored procedures to dynamically loaded libraries, or using an interpreted language like Lua/Python/JavaScript?

Re: Bedrock – Rock-solid distributed data

#94

Post by quinthar »

hbrundage wrote: post #75
quinthar wrote: post #28

Earlier quoted context omitted.

Thanks for asking! Re: "SQLite can be beat really easily by the client/server systems since each SQLite instance only supports one concurrent writer" -- That's not actually true, sqlite supports concurrent writers via their page-locking branch (and changesets allow for effectively row-level locks). But none of that matters, because single-threaded replication means any multi-threaded write capability is irrelevant. R…

Don't let me barge in and tell you how to do your job or anything, but for a piece of a system designed and marketed as the bottom, super stable piece of the stack, shipping out a brand new technology not yet in trunk seems kind of dangerous, no? The MVCC implementations in your competitors have been battle tested for stability and maximum performance for decades. Gonna be hard to compete with that. Re: "struggles ov…

Lots of comments! I'll try to address the key points:

Re: "brand new technology not yet in trunk" -- I'm not sure what you mean by that. Bedrock has been used continuously for 8 years.

Re: "struggles over high-latency, low-reliability WAN connections" -- Ah, I mean MySQL's replication is designed for active/passive deployments with manual failover connected via fast/reliable networks. Not saying it can't support slow/unreliable WAN connections, only that it requires a lot of glue code and manual recovery when things go more wrong thn it's designed to handle.

Re: "the engineering required to correctly implement this distributed system is genuinely challenging" -- Agreed! This is precisely what Bedrock provides.

Re: "SQL fanout" -- To my knowledge, no RDBMS does this automatically. Furthermore, very few real world applications actually require this. Don't get me wrong: this is cool stuff. But this isn't a common requirement. Most businesses will never exceed the capacity of a single server -- the number of businesses that truly require this level of scalability is very small.

Re: "I just don't understand what niche Bedrock is trying to fill." -- The niche of businesses that want a simple SQL database that has built in automatic failover.

Re: "multiple logical SQL instances" -- I'm not sure what you mean by this. Bedrock maintains a single contiguous SQL database across all nodes.

Re: "you only get as much read throughput for as much data as you are willing to duplicate" -- Yes, I'm saying there are very few reasons not to duplicate it all for the vast majority of real world use cases.

Thanks!

Re: Bedrock – Rock-solid distributed data

#95

Post by quinthar »

ngrilly wrote: post #92
quinthar wrote: post #84

Earlier quoted context omitted.

Every write transaction ever committed to the database is assigned a unique ID, and the most recent few million transactions are kept in a "journal" table. This is what allows nodes to figure out what happened while they were offline and re-execute the missing transactions. A side-effect of that is we know exactly how many write transactions have ever been committed.

That's a funny side-effect! Should I understand the transaction unique ID is an incrementing number?

Correct!

Re: Bedrock – Rock-solid distributed data

#96

Post by quinthar »

Jweb_Guru wrote: post #86
Jweb_Guru wrote: post #64

Earlier quoted context omitted.

The question is: (1) In the event of a crash, how does it know which transactions it's missing without a total order on the write transactions? (2) In the non-failure case, how do you know the transactions in one batch are ordered before the transactions in a subsequent batch (which requires the replica to identify any "gaps" within the previous batch so it can wait for them to come in before continuing on)? I don't…

Also, as pointed out in https://twitter.com/aphyr/status/788757992829222912 , even your current solution isn't safe if your commit order isn't total across leader changes (you can resolve this by adding an epoch number that increments every time leader election occurs). Strongly recommend you take him up on his offer.

To be clear, we're talking about functionality that is not implemented or fully designed. Today all transactions are committed on all nodes in the same order, which is a much simpler world. I agree, the multi-threaded replication case is a much more complex and interesting world, with much greater performance opportunities. Lots of exciting problems to solve when we get there!

Re: Bedrock – Rock-solid distributed data

#97

Post by Jweb_Guru »

quinthar wrote: post #96
Jweb_Guru wrote: post #86

Earlier quoted context omitted.

Also, as pointed out in https://twitter.com/aphyr/status/788757992829222912 , even your current solution isn't safe if your commit order isn't total across leader changes (you can resolve this by adding an epoch number that increments every time leader election occurs). Strongly recommend you take him up on his offer.

To be clear, we're talking about functionality that is not implemented or fully designed. Today all transactions are committed on all nodes in the same order, which is a much simpler world. I agree, the multi-threaded replication case is a much more complex and interesting world, with much greater performance opportunities. Lots of exciting problems to solve when we get there!

> Today all transactions are committed on all nodes in the same order, which is a much simpler world.

This is difficult to reconcile with:

> - For the highest performance, you can designate a transaction as "asynchronous" and the master will commit immediately

because if the leader crashes, a replica becomes leader and starts accepting writes, then the old leader recovers as a replica, without something like an epoch number it won't be able to tell that it has commits that the current leader doesn't (using a unique incrementing transaction number based on just a counter at the leader won't work, because it won't necessarily be unique across leader elections thanks to the asynchronous commits).

Re: Bedrock – Rock-solid distributed data

#99

Post by quinthar »

ngrilly wrote: post #93
quinthar wrote: post #87

Earlier quoted context omitted.

Great questions! Yes, indexing is actually a significant challenge at our scale, as it can take a long time to add and currently, it is a blocking operation. However, the sqlite team is amazing and has all sorts of write-concurrency tricks up their sleeve to allow for creating indexes in a parallel thread. We haven't used it yet, but we plan to. As for the stored procedures, yes they are written in C++, and thus depl…

> Yes, indexing is actually a significant challenge at our scale, as it can take a long time to add and currently, it is a blocking operation. What do you do during index creation: Do you drop write requests on the floor? Do you store them in a persistent queue somewhere? Or do you remove a node from the cluster, create the index, re-add the node to the cluster, and repeat this on every other nodes? I'm asking becaus…

> What do you do during index creation: Do you drop write requests on the floor? Do you store them in a persistent queue somewhere? Or do you remove a node from the cluster, create the index, re-add the node to the cluster, and repeat this on every other nodes?

The last of those -- for small indexes, we just replicate them out like normal queries. For large indexes, we take that node down and add offline.

> So, basically, you're doing a rolling upgrade? Have you evaluated compiling your stored procedures to dynamically loaded libraries, or using an interpreted language like Lua/Python/JavaScript?

Correct, rolling upgrades. It's worked well to date, but the idea of putting the plugin into a dynamically loaded library is really interesting. Our plugin system is relatively new (only the past few months) so it hasn't really been considered. Great idea!

Re: Bedrock – Rock-solid distributed data

#100

Post by quinthar »

Jweb_Guru wrote: post #97
quinthar wrote: post #96

Earlier quoted context omitted.

To be clear, we're talking about functionality that is not implemented or fully designed. Today all transactions are committed on all nodes in the same order, which is a much simpler world. I agree, the multi-threaded replication case is a much more complex and interesting world, with much greater performance opportunities. Lots of exciting problems to solve when we get there!

> Today all transactions are committed on all nodes in the same order, which is a much simpler world. This is difficult to reconcile with: > - For the highest performance, you can designate a transaction as "asynchronous" and the master will commit immediately because if the leader crashes, a replica becomes leader and starts accepting writes, then the old leader recovers as a replica, without something like an epoch…

Ah, sorry for the confusion. Every transaction is given an incrementing ID by the leader, and every follower commits the transactions in ID order.

Furthermore, every commit has a running SHA hash of all prior commits (and every node keeps a history of the last few million commits). This way any two nodes can compare their journals to make sure they agree -- and if there is any split, then the cluster kicks that node out.

Basically, there is no scenario in which a node that commits a different transaction (or a transaction in a different order) is allowed to remain in the cluster.

Post reply on HN