Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Marmot: Multi-writer distributed SQLite based on NATS

github.com

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Marmot: Multi-writer distributed SQLite based on NATS

#21

Post by dilyevsky »

latchkey wrote: post #7

I can totally see some junior dev who doesn't fully grok transactions (and not reading/understanding the "What happens when there is a race condition?" section), trying to deploy something like this and scratching their head why it falls over completely in production. The site worked fine when it was just me testing it! This is a neat proof of concept and I encourage experimentation. But, if you're developing somethi…

If you let juniors design and deploy a DB layer to prod it's your fault not the DBs. Transaction isolation is generally complicated topic that a lot of senior devs have a pretty tenuous grasp on ime and distributed Postgres solutions don't solve this particularly well either last I checked

Re: Marmot: Multi-writer distributed SQLite based on NATS

#22

Post by KRAKRISMOTT »

summarity wrote: post #12

If you're interested in this, here are some related projects that all take slightly different approaches: - LiteSync directly competes with Marmot and supports DDL sync, but is closed source commercial (similar to SQLite EE): https://litesync.io - dqlite is Canonical's distributed SQLite that depends on c-raft and kernel-level async I/O: https://dqlite.io - cr-sqlite is a Rust-based loadable extension that adds CRDT…

Don't forget Turso's libsql which uses a local node+reconciliation

https://medium.com/chiselstrike/introducing-embedded-replica...

Re: Marmot: Multi-writer distributed SQLite based on NATS

#23

Post by latchkey »

dilyevsky wrote: post #21
latchkey wrote: post #7

I can totally see some junior dev who doesn't fully grok transactions (and not reading/understanding the "What happens when there is a race condition?" section), trying to deploy something like this and scratching their head why it falls over completely in production. The site worked fine when it was just me testing it! This is a neat proof of concept and I encourage experimentation. But, if you're developing somethi…

If you let juniors design and deploy a DB layer to prod it's your fault not the DBs. Transaction isolation is generally complicated topic that a lot of senior devs have a pretty tenuous grasp on ime and distributed Postgres solutions don't solve this particularly well either last I checked

> If you let juniors design and deploy a DB layer to prod it's your fault not the DBs.

Some companies only have juniors. 28 years ago, I was the junior at my company, and the first/only engineer.

Let's also be real here, most applications don't need distributed Postgres either and those that do, will have senior engineers on staff.

Re: Marmot: Multi-writer distributed SQLite based on NATS

#24

Post by spiffytech »

summarity wrote: post #12

If you're interested in this, here are some related projects that all take slightly different approaches: - LiteSync directly competes with Marmot and supports DDL sync, but is closed source commercial (similar to SQLite EE): https://litesync.io - dqlite is Canonical's distributed SQLite that depends on c-raft and kernel-level async I/O: https://dqlite.io - cr-sqlite is a Rust-based loadable extension that adds CRDT…

Also Expensify's Bedrock, which powers their widely-circulated "Scaling SQLite to 4M QPS" article:

https://bedrockdb.com/

https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

Re: Marmot: Multi-writer distributed SQLite based on NATS

#25

Post by maxpert »

Author of Marmot here. Marmot was born out of my own use-case (was building replicated SQLite based cache). While working on various problems I realized how well suited it might be for read heavy sites/workloads. If I take a typical CMS site 90% of the time it's just reading and SQLite is perfect for that, but then how I get independently deployed nodes to replicate data. The philosophy I am sticking to so far:

- Sidecar! I would avoid any kind of in process library at any cost. Call me biased but I don't trust someone's code in my process space causing it to crash.

- No master - each node should be able to make progress on its own, if these processes go down your own process will keep functioning. They will converge once everything is back.

- Easy to start, yet hard to master - You can get up and running pretty quickly, but make no mistake this tool is not for rookie who doesn't understand how incremental primary keys are bad, and how to they can keep things conflict free.

I am far from getting everything I need in there, and again my philosophy might evolve over time as well. Talking to people on Discord has helped me think through use-cases a lot, so keep the good feedback coming. Would love to answer any questions people might have here.

Re: Marmot: Multi-writer distributed SQLite based on NATS

#26

Post by maxpert »

latchkey wrote: post #7

I can totally see some junior dev who doesn't fully grok transactions (and not reading/understanding the "What happens when there is a race condition?" section), trying to deploy something like this and scratching their head why it falls over completely in production. The site worked fine when it was just me testing it! This is a neat proof of concept and I encourage experimentation. But, if you're developing somethi…

I completely agree with your analysis. Understanding the complexities of achieving convergence with basic auto-increment counters without advanced CRDT types is 101 IMO. Those familiar with these issues inherently comprehend the challenges involved. While it's plausible for someone to leverage a library atop Marmot to construct and synchronize such types, it's important to note that this tool isn't tailored for junior developers grappling with transactional intricacies. I've witnessed instances where inexperienced developers initiate transactions and make HTTP calls while holding locks, resulting in system outages. Marmot isn't intended for individuals lacking a solid understanding of distributed systems. My recommendation aligns with advising entry-level individuals to explore these tools only when they reach a scale where such complexities become pertinent.

Re: Marmot: Multi-writer distributed SQLite based on NATS

#27

Post by maxpert »

simonw wrote: post #2

Something that wasn't clear to me from the README: how does this handle duplicate IDs? If I have a table with a string primary key and I insert a row in one node with ID "hello" and do the same thing (but with different column data) on another node, what happens?

Glad you asked the question. I never recommend use auto-incrementing IDs in production always generate one (e.g. Twitter Snowflake). With ID generators you get rid of collisions.

Re: Marmot: Multi-writer distributed SQLite based on NATS

#29

Post by wtatum »

I see a lot of projects started in this space and all of them appear to have multi-writer as a goal. I've been interested for a long time (and have started and stopped) in a solution for single-write multi-read with eventual consistency. I chatted with @benbjohnson on a LiteStream ticket about the possibility of adding a mobile client to receive the replicas to mobile devices but I think that option isn't really consistent with the new direction of that work for LiteFS at Fly.

To me the multi-writer "collaborative" use case is super powerful but also has a lot of challenges. I personally would see a lot of value in a solution for eventually consistent read-replicas that are available for end-client (WASM or mobile native) replication but still funnel updates through traditional APIs with store-and-forward for the offline case.

Is anybody aware of an open-source project pursuing that goal that maybe I haven't come across?

Re: Marmot: Multi-writer distributed SQLite based on NATS

#30

Post by meepmorp »

simonw wrote: post #10
governmentdude wrote: post #6

Earlier quoted context omitted.

In what context would you use a single string as the ID that would be subject to collisions (instead of including a user id as part of the ID)?

People do all sorts of weird things with databases, and if you want to run existing software against a database (as opposed to greenfield development deliberately targeting Marmot) you need to understand what will happen in different cases.

> if you want to run existing software against a database (as opposed to greenfield development deliberately targeting Marmot) you need to understand what will happen in different cases.

A great point, but from the readme in the Marmot repo:

> It does not require any changes to your existing SQLite application logic for reading/writing.

I suspect you’re probably still right, but that’s not what the author claims.

Post reply on HN