Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Marmot: Multi-writer distributed SQLite based on NATS

github.com

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Marmot: Multi-writer distributed SQLite based on NATS

#41

Post by otoolep »

zellyn wrote: post #40
summarity wrote: post #12

If you're interested in this, here are some related projects that all take slightly different approaches: - LiteSync directly competes with Marmot and supports DDL sync, but is closed source commercial (similar to SQLite EE): https://litesync.io - dqlite is Canonical's distributed SQLite that depends on c-raft and kernel-level async I/O: https://dqlite.io - cr-sqlite is a Rust-based loadable extension that adds CRDT…

Is there a system where a single thread writes (ala David Crawshaw's “One Process Programming -- https://crawshaw.io/blog/one-process-programming-notes ), and then changes are replicated out quickly and efficiently? That plus a globally incrementing version number is what I want…

That's a (very) simple way to describe what rqlite mostly does (and similar systems, many of which are listed below), though it depends on how "quickly" you want it. If you want speed, it's hard to do it fast -- though accepting a small possibility of data loss will get you much faster performance. rqlite also offers that, via Queued Writes[1].

[1] https://rqlite.io/docs/api/queued-writes/

Re: Marmot: Multi-writer distributed SQLite based on NATS

#42

Post by latchkey »

dilyevsky wrote: post #38
latchkey wrote: post #23

Earlier quoted context omitted.

> If you let juniors design and deploy a DB layer to prod it's your fault not the DBs. Some companies only have juniors. 28 years ago, I was the junior at my company, and the first/only engineer. Let's also be real here, most applications don't need distributed Postgres either and those that do, will have senior engineers on staff.

I’m aware such places exist and my point still stands. By your own logic they dont need anything beyond webflow, squarespace or the like and shoudnt be messing with dbs (and this includes postgres)

By your own logic, nobody becomes senior through experience.

Re: Marmot: Multi-writer distributed SQLite based on NATS

#43

Post by dilyevsky »

latchkey wrote: post #42
dilyevsky wrote: post #38

Earlier quoted context omitted.

I’m aware such places exist and my point still stands. By your own logic they dont need anything beyond webflow, squarespace or the like and shoudnt be messing with dbs (and this includes postgres)

By your own logic, nobody becomes senior through experience.

You generally become senior by getting exposure and proficiency with lots of different concepts which is the opposite to the spirit of your original comment here (hence the downvotes, I suspect). Only going with what everyone else is doing is how you become junior N times over.

Re: Marmot: Multi-writer distributed SQLite based on NATS

#44

Post by stocknear »

rmbyrro wrote: post #37
stocknear wrote: post #17

I'm using Marmot for my own website on production. Up to this date there were no problems. If I had any technical issues (i.e. questions, optimizations etc) I always asked the developer maxpert and he gave me in-depth answers that helped me personally a lot. In my case I have much love for Marmot and hopefully it grows and helps a bigger community

What scale does you site operate in terms of QPS?

Not much at the current stage. In range of 50-500 QPS.

However, my main goal for Marmot was to decrease the latency of my website by putting the db as close as I can to the end user

Re: Marmot: Multi-writer distributed SQLite based on NATS

#45

Post by rmbyrro »

stocknear wrote: post #44
rmbyrro wrote: post #37

Earlier quoted context omitted.

What scale does you site operate in terms of QPS?

Not much at the current stage. In range of 50-500 QPS. However, my main goal for Marmot was to decrease the latency of my website by putting the db as close as I can to the end user

50-500 QPS is quite substancial scale.

It's about +5M requests per day or ~2B per year. That's on the lower end! The higher end is ~20B requests per year!

I think using distributed SQL for that kind of scale may not be a wise choice, but that's my opinion...

Re: Marmot: Multi-writer distributed SQLite based on NATS

#46

Post by maxpert »

latchkey wrote: post #32
maxpert wrote: post #26

Earlier quoted context omitted.

I completely agree with your analysis. Understanding the complexities of achieving convergence with basic auto-increment counters without advanced CRDT types is 101 IMO. Those familiar with these issues inherently comprehend the challenges involved. While it's plausible for someone to leverage a library atop Marmot to construct and synchronize such types, it's important to note that this tool isn't tailored for junio…

> My recommendation aligns with advising entry-level individuals to explore these tools only when they reach a scale where such complexities become pertinent. I posit that if you're at that scale, you're figuring out how to get distributed postgres to work and not messing with things like Marmot.

Depends on use-case. What you are building, here is one use-case for example where it made sense over postgres https://www.youtube.com/watch?v=HycGtLjlikI

Re: Marmot: Multi-writer distributed SQLite based on NATS

#47

Post by maxpert »

jensneuse wrote: post #19

This is really cool. We've just created a POC that bridges Federated GraphQL Subscriptions and NATS, so this could maybe work together? Here's a small video of combining Federated Subscriptions and event driven architecture through NATS: https://twitter.com/TheWorstFounder/status/17341349261133783... ?

Would absolutely love to explore it deeper. Feel free to drop by community discord if you start experimenting with Marmot.

Re: Marmot: Multi-writer distributed SQLite based on NATS

#48

Post by latchkey »

maxpert wrote: post #46
latchkey wrote: post #32

Earlier quoted context omitted.

> My recommendation aligns with advising entry-level individuals to explore these tools only when they reach a scale where such complexities become pertinent. I posit that if you're at that scale, you're figuring out how to get distributed postgres to work and not messing with things like Marmot.

Depends on use-case. What you are building, here is one use-case for example where it made sense over postgres https://www.youtube.com/watch?v=HycGtLjlikI

Ok, I watched that. I wasn't able to understand what the actual use-case was other than a demo of setting this up in an infinitely over complicated system.

Re: Marmot: Multi-writer distributed SQLite based on NATS

#49

Post by [deleted] »

dilyevsky wrote: post #43
latchkey wrote: post #42

Earlier quoted context omitted.

By your own logic, nobody becomes senior through experience.

You generally become senior by getting exposure and proficiency with lots of different concepts which is the opposite to the spirit of your original comment here (hence the downvotes, I suspect). Only going with what everyone else is doing is how you become junior N times over.

[deleted]

Re: Marmot: Multi-writer distributed SQLite based on NATS

#50

Post by latchkey »

dilyevsky wrote: post #43
latchkey wrote: post #42

Earlier quoted context omitted.

By your own logic, nobody becomes senior through experience.

You generally become senior by getting exposure and proficiency with lots of different concepts which is the opposite to the spirit of your original comment here (hence the downvotes, I suspect). Only going with what everyone else is doing is how you become junior N times over.

> Only going with what everyone else is doing is how you become junior N times over.

Wild thought, this is not my experience at all. I don't even understand how you can come to the conclusion.

Post reply on HN