Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bedrock – Rock-solid distributed data

bedrockdb.com

Post reply on HN
Print view View original

81–90 of 110 posts

Re: Bedrock – Rock-solid distributed data

#81

Post by Jweb_Guru »

spotman wrote: post #79
quinthar wrote: post #48

Earlier quoted context omitted.

I should get better stats. 4.5M users, 6 nodes split between 3 datacenters. Each node has 16 CPUs. 2158341079 total write transactions (over 8 years); not sure how many read (10-100x more?). I'll try to get better stats on peak read/write transactions per second. Not sure the total number of rows of the largest table (that actually takes a long time to count). Counting "failures" is difficult -- if it breaks, it's be…

2 billion writes over 8 years? Cool if so, but many database systems both MySQL, and others are capable of getting this in a day. I know this because I maintain some high availability MySQL systems that see about 4 billion writes per day across a similar amount of hardware. Have you run this through Jespen or done any actual load testing or deep testing for failures related to machines dying, network partitions or ot…

While I agree that we're not talking about huge write volume in the grand scheme of things, and I definitely agree with you that Bedrock should pay aphyr to run this through Jepsen (particularly with their new replication strategy--see some of my concerns below), Expensify's overall transaction volume is probably close to the limit of what 95% of businesses are ever going to see, they have contractual latency and availability requirements they have to fulfill that are also probably more stringent than 95% of businesses are ever going to see, and they've been doing it with a single master database (no partitioning) and a single writer on that master. I think it's useful information that SQLite works for them, because it means SQLite would work fine for most people and (as is evidenced by this thread) a lot of people don't think SQLite can work for a site like Expensify.

And seeing actual numbers are important, too. People often act like in terms of write volume you're either shared hosting fodder, or you're Facebook, but there's a lot of room in between, and a lot of HA workloads are very read-dominated. It's very hard to find information about the businesses fall in the middle of that spectrum because usually no technology company talks about its traffic unless it's bragging about it (and then will release only vague information that could be spun in lots of different ways--for instance, not to call you out, I have no clue what "HA" means in your context, whether you need distributed transactions, whether the writes are cleanly partitionable, what isolation level you need, what your latency requirements are, or what the read:write ratio looks like). So I'd rather encourage people to release their numbers than poo-poo them because they aren't as big as whatever the biggest system you've worked on is.

Re: Bedrock – Rock-solid distributed data

#82

Post by spotman »

Jweb_Guru wrote: post #81
spotman wrote: post #79

Earlier quoted context omitted.

2 billion writes over 8 years? Cool if so, but many database systems both MySQL, and others are capable of getting this in a day. I know this because I maintain some high availability MySQL systems that see about 4 billion writes per day across a similar amount of hardware. Have you run this through Jespen or done any actual load testing or deep testing for failures related to machines dying, network partitions or ot…

While I agree that we're not talking about huge write volume in the grand scheme of things, and I definitely agree with you that Bedrock should pay aphyr to run this through Jepsen (particularly with their new replication strategy--see some of my concerns below), Expensify's overall transaction volume is probably close to the limit of what 95% of businesses are ever going to see, they have contractual latency and ava…

My point is that if they are advertising rock solid distributed data, I don't think they have achieved that without putting it through some more paces.

I agree, actual numbers are great.

We also agree, jespen is great.

But I am saying, another path, to a similar level of confidence jespen provides, is some real critical stress testing.

Sure, it will be out of the scope of what most people need, but it will give them the confidence to say "rock solid distributed data".

Re: Bedrock – Rock-solid distributed data

#84

Post by quinthar »

ngrilly wrote: post #70
quinthar wrote: post #29

Earlier quoted context omitted.

Re: SQLite's performance -- Can you point to anything that shows SQLite isn't the fastest for some class of query? In my experience (though admittedly I haven't formally benchmarked) it's at least as fast, and generally faster (due to the dramatically reduced overhead to query it over a traditional database server). Re: SQLite reliability. I'm confused how SQLite isn't more reliable than every other contender, merely…

> 2158341079 writes I'm curious: how do you keep track of this number?

Every write transaction ever committed to the database is assigned a unique ID, and the most recent few million transactions are kept in a "journal" table. This is what allows nodes to figure out what happened while they were offline and re-execute the missing transactions. A side-effect of that is we know exactly how many write transactions have ever been committed.

Re: Bedrock – Rock-solid distributed data

#85

Post by quinthar »

harryf wrote: post #65

Hmmm under plugins... * Jobs - Provides a simple job queue. * Cache - Provides a simple replicated cache. ...while these look like nice-to-have features which make life easy for developers, IMO having a database which is also a queue and is also a cache is a way to end up with a mess you can't scale out and is hard to reason about. Have seen this happen with Redis - unless you're disciplined about how you use it and…

Ya, I'm not sure I agree with that. I understand that argument in theory, but in practice the Jobs and Cache plugins have like 1% as much code as the Bedrock core -- and use all of it. So the idea that it makes sense to have a separate database, job queue, and cache -- despite all doing 99% of the same work -- strikes me as rather redundant and needlessly complex.

In practice, a job queue, a cache, and a database all do largely the same thing: maintain some internal state on disk, respond to some networked API calls, and synchronize across multiple servers. The actual logic of a job queue or cache is pretty trivial compared to what's underneath.

Indeed, I think the lesson of Redis is exactly why you should build these systems on top of a real database: they are struggling to hack on replication and reliable storage, even though these are "free" when built atop a database.

Re: Bedrock – Rock-solid distributed data

#86

Post by Jweb_Guru »

Jweb_Guru wrote: post #64
quinthar wrote: post #63

Earlier quoted context omitted.

To be clear, my sense is that every replica would receive every transaction, and would be free to commit the transactions inside each batch in any order. Now I do agree that it's tricky to avoid "gaps" in the failure cases. However, every replica keeps a record of the past several million transactions (we aim for 3 days), and every transaction is assigned a unique ID. When a replica starts up, it "synchronizes" down…

The question is: (1) In the event of a crash, how does it know which transactions it's missing without a total order on the write transactions? (2) In the non-failure case, how do you know the transactions in one batch are ordered before the transactions in a subsequent batch (which requires the replica to identify any "gaps" within the previous batch so it can wait for them to come in before continuing on)? I don't…

Also, as pointed out in https://twitter.com/aphyr/status/788757992829222912, even your current solution isn't safe if your commit order isn't total across leader changes (you can resolve this by adding an epoch number that increments every time leader election occurs). Strongly recommend you take him up on his offer.

Re: Bedrock – Rock-solid distributed data

#87

Post by quinthar »

ngrilly wrote: post #77

Two questions: If I'm not mistaken, creating an index is a blocking operation in SQLite (other writers are blocked). How do you manage this in production? How do you create a stored procedure? Do you have to recompile the whole program? How do you deploy with zero downtime?

Great questions! Yes, indexing is actually a significant challenge at our scale, as it can take a long time to add and currently, it is a blocking operation. However, the sqlite team is amazing and has all sorts of write-concurrency tricks up their sleeve to allow for creating indexes in a parallel thread. We haven't used it yet, but we plan to.

As for the stored procedures, yes they are written in C++, and thus deploying them requires compiling and upgrading the server itself. However, we have 6 of them (and honestly, any one of which has enough read capacity to satisfy about all our traffic) so a minute of downtime to upgrade for each independently isn't a problem.

The result is zero downtime as perceived by the user, even though each server has occasional downtime for maintenance and upgrades. In general I'd say we upgrade the database about weekly, or more frequent depending on how active we are in the stored procedures.

Re: Bedrock – Rock-solid distributed data

#88

Post by quinthar »

petre wrote: post #67

How is data distributed over multiple hosts? Aside from being replicated, is is sharded? Does it have any sort of redundancy? What happens if you lose a host?

Every host has all data. (This sounds crazy, until you do the math on how cheap storage is and how comparatively "little" data people actually need in the real world.) So we have 6x redundancy in normal operation, across three datacenters (and three different power grids, three different providers). There are also nightly backups.

Re: Bedrock – Rock-solid distributed data

#90

Post by quinthar »

msackman wrote: post #80
quinthar wrote: post #43

Earlier quoted context omitted.

Re: SQLite -- Yes, Bedrock is just the replication layer, SQLite does all the SQL and storage. Re: Paxos -- It's our own implementation. Split brain is prevented by the master refusing to stand up unless a majority of configured (not just active) slaves approve its standup request. So in a 6 node deployment, 1 master and 5 slaves, 3 of the slaves would need to approve. This means one "half" would have 4 nodes, and th…

Paxos: I've been trying to look for that. Having cloned the code and grepped for paxos I'm getting no hits. Where is the paxos implementation?

It's not called out specifically (actually, when writing it I didn't even know what Paxos was, and only realized I had implemented it years later). However, the logic is here: https://github.com/Expensify/Bedrock/blob/master/sqliteclust...
Post reply on HN