Live data from Hacker News

Marmot: Multi-writer distributed SQLite based on NATS

github.com

11–20 of 50 posts

Re: Marmot: Multi-writer distributed SQLite based on NATS

#11
post #2

Something that wasn't clear to me from the README: how does this handle duplicate IDs? If I have a table with a string primary key and I insert a row in one node with ID "hello" and do the same thing (but with different column data) on another node, what happens?

In what context would you use a single string as the ID that would be subject to collisions (instead of including a user id as part of the ID)?

A better example might be a field that must be unique, like a URL slug. Suppose you have a database of products and the product table has a field used for URLs, like “/tshirt” and “/sweater”. You need those to be unique within the table.

The question remains - how does Marmot enforce a uniqueness constraint? If you don’t like the product example, fine, but it is easy to think of others. It would be unfortunate if marmot is incapable of supporting uniqueness.

Re: Marmot: Multi-writer distributed SQLite based on NATS

#12
If you're interested in this, here are some related projects that all take slightly different approaches:

- LiteSync directly competes with Marmot and supports DDL sync, but is closed source commercial (similar to SQLite EE): https://litesync.io

- dqlite is Canonical's distributed SQLite that depends on c-raft and kernel-level async I/O: https://dqlite.io

- cr-sqlite is a Rust-based loadable extension that adds CRDT changeset generation and reconciliation to SQLite: https://github.com/vlcn-io/cr-sqlite

Slightly related but not really (no multi writer, no C-level SQLite API or other restrictions):

- comdb2 (Bloombergs multi-homed RDBMS using SQLite as the frontend)

- rqlite: RDBMS with HTTP API and SQLite as the storage engine, used for replication and strong consistency (does not scale writes)

- litestream/LiteFS: disaster recovery replication

- liteserver: active read-only replication (predecessor of LiteSync)

Re: Marmot: Multi-writer distributed SQLite based on NATS

#13

Earlier quoted context omitted.

In what context would you use a single string as the ID that would be subject to collisions (instead of including a user id as part of the ID)?

A better example might be a field that must be unique, like a URL slug. Suppose you have a database of products and the product table has a field used for URLs, like “/tshirt” and “/sweater”. You need those to be unique within the table. The question remains - how does Marmot enforce a uniqueness constraint? If you don’t like the product example, fine, but it is easy to think of others. It would be unfortunate if mar…

As I understand it, transactions are still serialized for any given table, just not across tables. Wouldn’t that solve this uniqueness constraint issue?

Re: Marmot: Multi-writer distributed SQLite based on NATS

#14
post #3

Earlier quoted context omitted.

I'm not 100% sure of this but from the README section on race conditions: "the last writer will always win. This means there is NO serializability guarantee of a transaction spanning multiple tables. This is a design choice, in order to avoid any sort of global locking, and performance." So it sounds like in your example, whichever node writes last with a given primary key will be the data you'll end up with.

Based on their description of how it works with jetstream i think last insert will fail because it will see the row already exists in jetstream Within a single table they have tx serializability, just not with multiple tables

[deleted]

Re: Marmot: Multi-writer distributed SQLite based on NATS

#15
post #7

I can totally see some junior dev who doesn't fully grok transactions (and not reading/understanding the "What happens when there is a race condition?" section), trying to deploy something like this and scratching their head why it falls over completely in production. The site worked fine when it was just me testing it! This is a neat proof of concept and I encourage experimentation. But, if you're developing somethi…

There’s a reason that this is called “hacker news” and not “just use the industry standard for the last 3 decades news”.

Won’t downvote you for giving pragmatic advice, but I appreciate projects like this that slap together disparate technologies for an interesting goal, even if it isn’t the best choice for your usual Fortune 500 company.

Re: Marmot: Multi-writer distributed SQLite based on NATS

#16
post #7

I can totally see some junior dev who doesn't fully grok transactions (and not reading/understanding the "What happens when there is a race condition?" section), trying to deploy something like this and scratching their head why it falls over completely in production. The site worked fine when it was just me testing it! This is a neat proof of concept and I encourage experimentation. But, if you're developing somethi…

+1 for Postgres. It's actually ridiculous how much good software you can simply start using for free.

Re: Marmot: Multi-writer distributed SQLite based on NATS

#17
I'm using Marmot for my own website on production. Up to this date there were no problems.

If I had any technical issues (i.e. questions, optimizations etc) I always asked the developer maxpert and he gave me in-depth answers that helped me personally a lot.

In my case I have much love for Marmot and hopefully it grows and helps a bigger community

Re: Marmot: Multi-writer distributed SQLite based on NATS

#18
> In Marmot every row is uniquely mapped to a JetStream. This guarantees that for any node to publish changes for a row it has to go through same JetStream as everyone else.

and

> This means there is NO serializability guarantee of a transaction spanning multiple tables. This is a design choice, in order to avoid any sort of global locking, and performance.

But since the serialization happens at per row level, does this also mean no serializability guarantee of a transaction within a table too, not only spanning multiple tables?

Re: Marmot: Multi-writer distributed SQLite based on NATS

#19
This is really cool. We've just created a POC that bridges Federated GraphQL Subscriptions and NATS, so this could maybe work together? Here's a small video of combining Federated Subscriptions and event driven architecture through NATS: https://twitter.com/TheWorstFounder/status/17341349261133783...?

Re: Marmot: Multi-writer distributed SQLite based on NATS

#20

If you're interested in this, here are some related projects that all take slightly different approaches: - LiteSync directly competes with Marmot and supports DDL sync, but is closed source commercial (similar to SQLite EE): https://litesync.io - dqlite is Canonical's distributed SQLite that depends on c-raft and kernel-level async I/O: https://dqlite.io - cr-sqlite is a Rust-based loadable extension that adds CRDT…

- SQLSync: collaborative offline-first wrapper around SQLite
Post reply on HN