Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bedrock – Rock-solid distributed data

bedrockdb.com

Post reply on HN
Print view View original

101–110 of 110 posts

Re: Bedrock – Rock-solid distributed data

#101

Post by ngrilly »

quinthar wrote: post #99
ngrilly wrote: post #93

Earlier quoted context omitted.

> Yes, indexing is actually a significant challenge at our scale, as it can take a long time to add and currently, it is a blocking operation. What do you do during index creation: Do you drop write requests on the floor? Do you store them in a persistent queue somewhere? Or do you remove a node from the cluster, create the index, re-add the node to the cluster, and repeat this on every other nodes? I'm asking becaus…

> What do you do during index creation: Do you drop write requests on the floor? Do you store them in a persistent queue somewhere? Or do you remove a node from the cluster, create the index, re-add the node to the cluster, and repeat this on every other nodes? The last of those -- for small indexes, we just replicate them out like normal queries. For large indexes, we take that node down and add offline. > So, basic…

Great! Thanks for following-up.

If you have any link about "SQLite write-concurrency tricks to allow for creating indexes in a parallel thread", please share ;-)

Re: Bedrock – Rock-solid distributed data

#102

Post by dboreham »

quinthar wrote: post #53
bithavoc wrote: post #46

Earlier quoted context omitted.

Also with identical approach there is http://www.actordb.com

Oh, neat! That one seems to replace SQLite's storage engine with a custom one. But yes, also similar. Cool! Again, what is the largest real-world user? We've found a lot of the theories tend to break down when they are subjected to reality -- Bedrock's advantage is it's been hardened in the crucible of live production traffic for years.

ActorDB was developed for this service : http://www.emitcloud.com/home

Re: Bedrock – Rock-solid distributed data

#103

Post by quinthar »

dboreham wrote: post #102
quinthar wrote: post #53

Earlier quoted context omitted.

Oh, neat! That one seems to replace SQLite's storage engine with a custom one. But yes, also similar. Cool! Again, what is the largest real-world user? We've found a lot of the theories tend to break down when they are subjected to reality -- Bedrock's advantage is it's been hardened in the crucible of live production traffic for years.

ActorDB was developed for this service : http://www.emitcloud.com/home

Cool. It looks like they have 500-1000 Android installs. As context, Expensify has over 1M Android installs, and over 4M total users. So I think Bedrock is probably operating 4-5 orders of magnitude more activity, at least in this case.

Re: Bedrock – Rock-solid distributed data

#105

Post by justinclift »

quinthar wrote: post #90
msackman wrote: post #80

Earlier quoted context omitted.

Paxos: I've been trying to look for that. Having cloned the code and grepped for paxos I'm getting no hits. Where is the paxos implementation?

It's not called out specifically (actually, when writing it I didn't even know what Paxos was, and only realized I had implemented it years later). However, the logic is here: https://github.com/Expensify/Bedrock/blob/master/sqliteclust...

With these: https://github.com/Expensify/Bedrock/blob/ecda922dc279e06fda...

Do you use the Issue Tracker to keep on top of things like that, and/or prioritise?

Re: Bedrock – Rock-solid distributed data

#106

Post by Jweb_Guru »

quinthar wrote: post #100
Jweb_Guru wrote: post #97

Earlier quoted context omitted.

> Today all transactions are committed on all nodes in the same order, which is a much simpler world. This is difficult to reconcile with: > - For the highest performance, you can designate a transaction as "asynchronous" and the master will commit immediately because if the leader crashes, a replica becomes leader and starts accepting writes, then the old leader recovers as a replica, without something like an epoch…

Ah, sorry for the confusion. Every transaction is given an incrementing ID by the leader, and every follower commits the transactions in ID order. Furthermore, every commit has a running SHA hash of all prior commits (and every node keeps a history of the last few million commits). This way any two nodes can compare their journals to make sure they agree -- and if there is any split, then the cluster kicks that node…

I think this or something like this can probably work if you're okay with losing all data that wasn't acked by a majority (though I suspect actually recovering a divergent replica would be very difficult), but this doesn't work with the batch commit idea at all, does it? Seems like it enforces strict serial ordering of writes (even nonconflicting ones).

Re: Bedrock – Rock-solid distributed data

#107

Post by quinthar »

quinthar wrote: post #8
dsl wrote: post #5

For all the emphasis put on multi-datacenter replication, it would be awesome to see it documented! Really looking forward to playing with this. Cool stuff.

There's a bit of that here: http://bedrockdb.com/multizone.html But yes, lots more details to add over time. Thanks for asking!

And a bit more here: http://bedrockdb.com/synchronization.html

Re: Bedrock – Rock-solid distributed data

#108

Post by quinthar »

justinclift wrote: post #105
quinthar wrote: post #90

Earlier quoted context omitted.

It's not called out specifically (actually, when writing it I didn't even know what Paxos was, and only realized I had implemented it years later). However, the logic is here: https://github.com/Expensify/Bedrock/blob/master/sqliteclust...

With these: https://github.com/Expensify/Bedrock/blob/ecda922dc279e06fda... Do you use the Issue Tracker to keep on top of things like that, and/or prioritise?

We're using GitHub Issues: https://github.com/Expensify/Bedrock/issues However, honestly those specific issues aren't on the list. In general we focus less on what could happen, and more on what actually does happen. Those specific issues haven't ever occurred, and thus never got "fixed" because they never became real problems. But PRs welcome!!

Re: Bedrock – Rock-solid distributed data

#109

Post by justinclift »

Jweb_Guru wrote: post #81
spotman wrote: post #79

Earlier quoted context omitted.

2 billion writes over 8 years? Cool if so, but many database systems both MySQL, and others are capable of getting this in a day. I know this because I maintain some high availability MySQL systems that see about 4 billion writes per day across a similar amount of hardware. Have you run this through Jespen or done any actual load testing or deep testing for failures related to machines dying, network partitions or ot…

While I agree that we're not talking about huge write volume in the grand scheme of things, and I definitely agree with you that Bedrock should pay aphyr to run this through Jepsen (particularly with their new replication strategy--see some of my concerns below), Expensify's overall transaction volume is probably close to the limit of what 95% of businesses are ever going to see, they have contractual latency and ava…

Some of these FIXME's in Bedrock's er... Paxos implementation look like they could be important:

https://github.com/Expensify/Bedrock/blob/ecda922dc279e06fda...

Re: Bedrock – Rock-solid distributed data

#110

Post by alucab »

Hi this is very interesting. Can you elaborate on your concept of plugin development ?

You clearly state that your ideal model is to develop plugins that are embedded in the DB to execute core business functionalities. However, in the website there is only tutorial documentation about the SQL interface.

Is the only guide the code for now ?

Thanks !

Post reply on HN