Live data from Hacker News

We built a distributed DB capable of running across 100s of global locations

macrometa.co

11–20 of 29 posts

Re: We built a distributed DB capable of running across 100s of global locations

#11
Great work. Building a distributed database isn't easy at all and takes considerable effort. I'd like to see more on failure scenarios. Here's a few preliminary questions I have (admittedly, I haven't read the entire post, apologies if my queries are answered already):

What is the SLA for durability and availability of the db?

- how are scenarios like edge locations going down for multiple minutes handled? Are the writes lost?

- how many replicas of data at a single edge location?

- Are there limits on the table size (how does one enforce this in a multi-master mode)?

- what's the SLA on replication time to say, at least 50% edge locations, and then to 100%?

Are DDL operations allowed? If so, how are the conflicts handled?

If data is stored in LSMs underneath, how are geo location queries handled? Is there an index/materialised view? If so, how long does that take to generate?

What's the TPS/QPS supported for the KV interface?

- Is there a scenario where a set of operations at X TPS across edge locations might then take a long time to converge globally?

It'd be great if you can list down scenarios where the db can lose write to 'true conflicts' in a FAQ somewhere.

- For instance, what happens in a create-delete-create scenario where a table is created at one edge location, deleted at another, created at a third edge location; all the while when there's writes and reads happening globally?

Thanks.

Re: We built a distributed DB capable of running across 100s of global locations

#12
This is actually exciting. Most sales pitches for secret sauce services are much flimsier and don't ooze the actual technical possibilities and promises that this one does.

So, does it solve the double-spend problem or not? The prose is a bit ambiguous on that.

I don't want to work that out for myself. I want to be told one way or another, and perhaps given some simple examples like "users x and y both want to do z at roughly the same time; depending on several factors like the exact timings and network latencies, there are several outcomes. They are..." etc.

And of course we want to know what kind of realistic throughput and latency it gets, how easy it is to add and remove POPs (and what data gets lost when you do), etc.

Finally, it really needs to commission and publish a Jepsen analysis. Hasn't every database vendor learned how critical that is to winning the programming public's heart?

This seems to be a startup who is going to sell it as a service, and I hope them all the best. The DB world is full of better tech that didn't make the dent it technically deserved (hyperdex, tokudb, perhaps rethink etc) so a new startup has to be both vocal and embrace Jepsen from the beginning. I think HN is ripe for a new darling and whichever company becomes that darling might capture a lot of the ill-informed web-scale-before-we-have-a-customer market ;)

Re: We built a distributed DB capable of running across 100s of global locations

#14
post #2

The TL/DR on how: 1. Causal consistency with the ability to create collections (tables) with strict consistency - uses vector clocks and not wall clock timestamps for ordering DB operations 2. Streams to propagate DB changes from one geo location (node) to another with guaranteed ordering and reliable delivery 3. Generalized Operational CRDTs to make all DB operations Associative, Commutative, Idempotent and Distribu…

This is buzzword salad, not a coherent technical summary.

That's quite a harsh comment without anything substantial. It's really unfair to the parent comment who attempted to help the HN reader crowd with a summary.

Re: We built a distributed DB capable of running across 100s of global locations

#15
post #2

The TL/DR on how: 1. Causal consistency with the ability to create collections (tables) with strict consistency - uses vector clocks and not wall clock timestamps for ordering DB operations 2. Streams to propagate DB changes from one geo location (node) to another with guaranteed ordering and reliable delivery 3. Generalized Operational CRDTs to make all DB operations Associative, Commutative, Idempotent and Distribu…

This is buzzword salad, not a coherent technical summary.

I would say it’s fairly easy to read. Easier to read than the article. Don’t shoot the messenger.

Re: We built a distributed DB capable of running across 100s of global locations

#16
post #6

Shouldn't this be a "Show HN", since the submitter is the company CEO/President?

No

Show HN isn't for companies.

It's for broke developers trying to make a buck off the free advertising on HN and showcase their skills so that they get some employment based on meritocracy.

No company will use Show HN to showcase their work.

Re: We built a distributed DB capable of running across 100s of global locations

#17
post #2

The TL/DR on how: 1. Causal consistency with the ability to create collections (tables) with strict consistency - uses vector clocks and not wall clock timestamps for ordering DB operations 2. Streams to propagate DB changes from one geo location (node) to another with guaranteed ordering and reliable delivery 3. Generalized Operational CRDTs to make all DB operations Associative, Commutative, Idempotent and Distribu…

Translation:

There are two "modes" of operation of this product: "regular" and "SPOT collections".

In regular operation mode, the system will lose concurrent writes and cannot enforce ACID (the classical one) transactions. Your data is however highly available, eventually consistent and copied around the globe for faster access.

With SPOT collections you get ACID transactions but lose the distribution advantages. That is, the system's properties are akin to what you get with a postgres/mysql cluster.

It is an interesting product, but IMHO you should only go there if/when: your users are truly all over the world, and your system (or that part of it) requires few or no ACID transactions (think twice because this is a big one).

PS: This seems to be a "productified" version of the gun [1] database (not that this is a bad thing!). Am I right?

[1] https://github.com/amark/gun

Re: We built a distributed DB capable of running across 100s of global locations

#18
This looks pretty neat!

Considering that there's a FaaS layer too, which could be used along with the DB layer offering, what degree of vendor-lockin are users getting into?

Also, this product seems only targeted at those direly needing edge computing, thus missing out on many who might be using Firebase and Serverless (Lambda/Functions etc) for the works. Is that intentional?

Re: We built a distributed DB capable of running across 100s of global locations

#19
So, does it actually have CRDT operations? Doesn't seem like any of the interfaces expose them. Can you use a convergent counter, for example?

Also product page [1] has some weird claims. There is no such thing as "Strong Session Consistency", this is just another name for weak consistency guarantees. And Strong Eventual Consistency, which is a thing, requires using mergeable conflict-free operations that you don't seem to expose? The "smart operation ordering using intent prioritization" is also neither strong eventual consistency nor strong consistency.

[1] https://www.macrometa.co/product

Re: We built a distributed DB capable of running across 100s of global locations

#20

Great work. Building a distributed database isn't easy at all and takes considerable effort. I'd like to see more on failure scenarios. Here's a few preliminary questions I have (admittedly, I haven't read the entire post, apologies if my queries are answered already): What is the SLA for durability and availability of the db? - how are scenarios like edge locations going down for multiple minutes handled? Are the wr…

Thanks for the truly thoughtful questions - I will put together a short faq and post a link and a a tl/dr answer here.

We also have a more technical internals paper almost completed and will publish next week.

Thanks again! This is such a helpful comment.

Post reply on HN