Live data from Hacker News

Bedrock – Rock-solid distributed data

bedrockdb.com

71–80 of 110 posts

Re: Bedrock – Rock-solid distributed data

#71
post #47

Can u show stats of bedrock use in expensify? How many users? How many nodes? How many transactions per second? queries per second? DB size? largest table rowcount? node failures per month? how many times have u had a master failure? network partition incidents?

I should get better stats. 4.5M users, 6 nodes split between 3 datacenters. Each node has 16 CPUs. 2158341079 total write transactions (over 8 years); not sure how many read (10-100x more?). I'll try to get better stats on peak read/write transactions per second. Not sure the total number of rows of the largest table (that actually takes a long time to count). Counting "failures" is difficult -- if it breaks, it's be…

> 6 nodes split between 3 datacenters

This makes 2 nodes per datacenter. Do the 2 nodes cooperate in some way, or are they fully independent from each other (like shards)?

Re: Bedrock – Rock-solid distributed data

#72
post #69

Earlier quoted context omitted.

Thanks for asking! Re: "SQLite can be beat really easily by the client/server systems since each SQLite instance only supports one concurrent writer" -- That's not actually true, sqlite supports concurrent writers via their page-locking branch (and changesets allow for effectively row-level locks). But none of that matters, because single-threaded replication means any multi-threaded write capability is irrelevant. R…

> sqlite supports concurrent writers via their page-locking branch (and changesets allow for effectively row-level locks) I've been unable to find the branch on SQLite website. Do you have a link to the branch and changesets?

https://www.sqlite.org/src/timeline?n=100&r=begin-concurrent

Re: Bedrock – Rock-solid distributed data

#73

I think it is important to encourage people to build their own databases, so I want to commend you guys for doing it. But I do have some questions (disclosure: I am the author of https://github.com/amark/gun ): - Is it correct to say that Bedrock is primarily a replication layer for SQLite? Most things your homepage highlight seem to be SQLite features, not Bedrock features. - Elsewhere in this thread you mention you…

Re: SQLite -- Yes, Bedrock is just the replication layer, SQLite does all the SQL and storage. Re: Paxos -- It's our own implementation. Split brain is prevented by the master refusing to stand up unless a majority of configured (not just active) slaves approve its standup request. So in a 6 node deployment, 1 master and 5 slaves, 3 of the slaves would need to approve. This means one "half" would have 4 nodes, and th…

Is the set of servers that participate a paxos value too (ie determined using the consensus algorithm), or is it configured ? Can you grow a cluster without bringing it down ?

Re: Bedrock – Rock-solid distributed data

#74
post #71

Earlier quoted context omitted.

I should get better stats. 4.5M users, 6 nodes split between 3 datacenters. Each node has 16 CPUs. 2158341079 total write transactions (over 8 years); not sure how many read (10-100x more?). I'll try to get better stats on peak read/write transactions per second. Not sure the total number of rows of the largest table (that actually takes a long time to count). Counting "failures" is difficult -- if it breaks, it's be…

> 6 nodes split between 3 datacenters This makes 2 nodes per datacenter. Do the 2 nodes cooperate in some way, or are they fully independent from each other (like shards)?

Just found the answer to my question:

> In practice, we deploy two servers in each datacenter (across three geographically-distributed datacenters, so six nodes total in the cluster -- with one configured as a "permaslave" that doesn't participate in quorum). Given this, the node that's in the same datacenter generally gets most if not all of the master's transactions in a real world crash scenario.

Source: http://p2p-hackers.709552.n3.nabble.com/p2p-hackers-Advice-o...

Re: Bedrock – Rock-solid distributed data

#75

I don't quite get why SQLite is the right primitive to compose a large scale database system out of... If the design goal is write throughput, SQLite can be beat really easily by the client/server systems since each SQLite instance only supports one concurrent writer. I guess users of Bedrock could partition their datasets into different pieces that aren't often written in tandem, but why go through all the trouble i…

Thanks for asking! Re: "SQLite can be beat really easily by the client/server systems since each SQLite instance only supports one concurrent writer" -- That's not actually true, sqlite supports concurrent writers via their page-locking branch (and changesets allow for effectively row-level locks). But none of that matters, because single-threaded replication means any multi-threaded write capability is irrelevant. R…

Don't let me barge in and tell you how to do your job or anything, but for a piece of a system designed and marketed as the bottom, super stable piece of the stack, shipping out a brand new technology not yet in trunk seems kind of dangerous, no? The MVCC implementations in your competitors have been battle tested for stability and maximum performance for decades. Gonna be hard to compete with that.

Re: "struggles over high-latency, low-reliability WAN connections" If by struggle you mean a vanilla MySQL replication setup doesn't satisfy C, A and P of the CAP theorem then yeah I agree, however if you just mean that it struggles to be consistent with an unreliable network, that just means MySQL has made the same CAP tradeoff Bedrock has. AFAIK there aren't many critical bugs that mean you lose data that could have otherwise been stored in the presence of unreliable networks.

You're right that vanilla MySQL doesn't support automatic failover, however, some consider that a feature, because the engineering required to correctly implement this distributed system is genuinely challenging. Bedrock may have done it, but if there are some complicated cases that it has yet to address users may prefer to manage failover manually before they trust it to the system. MySQL also isn't the only system that we should draw comparison too -- Cassandra or Riak come to mind as well. Both have excellent replication for geo-redundancy and something akin to automatic failover by storing data in a configurable number of places. They don't support raw SQL statements like Bedrock does, but to be honest if Bedrock doesn't support actual SQL fanout where the results can be combined to make the thing seem like one big database, the client is left to do a bunch of the work like they would be doing in Cassandra to model their data and fetch it efficiently.

I'm not trying to suggest that MySQL or a MySQL orchestration layer is better than Bedrock for all use cases; I just don't understand what niche Bedrock is trying to fill. For basic consumers of the system it's more work to author your queries for Bedrock because (I think) you now have to manage multiple logical SQL instances instead of one virtual big one, and you only get as much read throughput for as much data as you are willing to duplicate. You only get write throughput for the same but at 1 writer per shard/slave/whatever it's called. It's using a homegrown replication algorithm that has the same consistency tradeoffs of other more widely deployed and thus better tested solutions, and it's not clear that database builders or the target audience for Bedrock-as-framework would actually want to use SQLite due to the performance considerations mentioned above.

Re: Bedrock – Rock-solid distributed data

#77
Two questions:

If I'm not mistaken, creating an index is a blocking operation in SQLite (other writers are blocked). How do you manage this in production?

How do you create a stored procedure? Do you have to recompile the whole program? How do you deploy with zero downtime?

Re: Bedrock – Rock-solid distributed data

#78
post #72
post #69

Earlier quoted context omitted.

> sqlite supports concurrent writers via their page-locking branch (and changesets allow for effectively row-level locks) I've been unable to find the branch on SQLite website. Do you have a link to the branch and changesets?

https://www.sqlite.org/src/timeline?n=100&r=begin-concurrent

Thanks! Is there some documentation already? Do you plan to integrate "BEGIN CONCURRENT" in the standard SQLite distribution?

Re: Bedrock – Rock-solid distributed data

#79
post #47

Can u show stats of bedrock use in expensify? How many users? How many nodes? How many transactions per second? queries per second? DB size? largest table rowcount? node failures per month? how many times have u had a master failure? network partition incidents?

I should get better stats. 4.5M users, 6 nodes split between 3 datacenters. Each node has 16 CPUs. 2158341079 total write transactions (over 8 years); not sure how many read (10-100x more?). I'll try to get better stats on peak read/write transactions per second. Not sure the total number of rows of the largest table (that actually takes a long time to count). Counting "failures" is difficult -- if it breaks, it's be…

2 billion writes over 8 years? Cool if so, but many database systems both MySQL, and others are capable of getting this in a day. I know this because I maintain some high availability MySQL systems that see about 4 billion writes per day across a similar amount of hardware.

Have you run this through Jespen or done any actual load testing or deep testing for failures related to machines dying, network partitions or other?

I encourage you to do so, because (not trying to sound rude) this kind of reads like "hey my homemade car has 25,000 miles on it and I still use it". But you can build a performance and failure testing framework to really put it through its paces.

Anyhow , have fun and good luck~~

Re: Bedrock – Rock-solid distributed data

#80

I think it is important to encourage people to build their own databases, so I want to commend you guys for doing it. But I do have some questions (disclosure: I am the author of https://github.com/amark/gun ): - Is it correct to say that Bedrock is primarily a replication layer for SQLite? Most things your homepage highlight seem to be SQLite features, not Bedrock features. - Elsewhere in this thread you mention you…

Re: SQLite -- Yes, Bedrock is just the replication layer, SQLite does all the SQL and storage. Re: Paxos -- It's our own implementation. Split brain is prevented by the master refusing to stand up unless a majority of configured (not just active) slaves approve its standup request. So in a 6 node deployment, 1 master and 5 slaves, 3 of the slaves would need to approve. This means one "half" would have 4 nodes, and th…

Paxos: I've been trying to look for that. Having cloned the code and grepped for paxos I'm getting no hits. Where is the paxos implementation?
Post reply on HN