11–20 of 66 posts
Ben and Matt - appreciate your contributions in this area. I'm interested in making contributions along with you. Please let me know if you are looking for help. Much Thanks.
Earlier quoted context omitted.
Very cool. I've been working on CRDTs for SQLite to allow for conflict-free replication. Its still super early but I have a prototype that works and, if you're up to it, I would love to pick your brain on the details of sqlite WALs and journaling. https://www.loom.com/share/0934f93364d340e0ba658146a974edb4
Right on, I always thought a CRDT SQLite database would be awesome. I thought about looking at the session extension[1] to handle the merges but I didn't get very far into it. Feel free to ping me at benbjohnson@yahoo.com if you want to chat WAL & journaling. I think it'd be difficult to implement CRDT at the journal level since it's all physical pages. Merging data in pages is going to sometimes cause splits and tha…
One perfectly reasonable use case for a read replica of a database is a bastion server. Database + web server on a machine that is firewalled both from the internet and from the business network. With read only access there is a much smaller blast radius if someone manages to compromise the machine.
The problem is that every single replication implementation I've seen expects the replicas to phone home to the master copy, not for the master copy to know of the replicas and stream updates to them. This means that your bastion machine needs to be able to reach into your LAN, which defeats half the point.
The most important question is, "what options exist to support this?" but I think the bigger question is why do we treat replicas as if they are full peers of the system of record when so often not only are they not - mechanically or philosophically - and in some cases couldn't be even if we wanted to? (eg, a database without multi-master support).
I have an adjacent problem, and I haven't been able to find anyone who has a fix for me. One perfectly reasonable use case for a read replica of a database is a bastion server. Database + web server on a machine that is firewalled both from the internet and from the business network. With read only access there is a much smaller blast radius if someone manages to compromise the machine. The problem is that every sing…
I believe Cassandra does not have a Master\follower architecture; it's following a ring based structure.
I have an adjacent problem, and I haven't been able to find anyone who has a fix for me. One perfectly reasonable use case for a read replica of a database is a bastion server. Database + web server on a machine that is firewalled both from the internet and from the business network. With read only access there is a much smaller blast radius if someone manages to compromise the machine. The problem is that every sing…
You can set up a PostgreSQL replica to be driven purely off of archive logs. It does not need direct access to the source database as it can pull the archive files via read only to a third location (e.g. file server or S3) that gets pushed by the source database server. The catch is that it will only be updated when an WAL file is pushed which can be driven either by size (automatically on an "active" database) or time (every N seconds or minutes). If you're fine with potentially being a minute behind the source, you can easily set this up.
LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.
LiteFS author here (also Litestream author). I'm happy to answer any questions folks have about how it works or what's on the roadmap.
I have an adjacent problem, and I haven't been able to find anyone who has a fix for me. One perfectly reasonable use case for a read replica of a database is a bastion server. Database + web server on a machine that is firewalled both from the internet and from the business network. With read only access there is a much smaller blast radius if someone manages to compromise the machine. The problem is that every sing…
I have an adjacent problem, and I haven't been able to find anyone who has a fix for me. One perfectly reasonable use case for a read replica of a database is a bastion server. Database + web server on a machine that is firewalled both from the internet and from the business network. With read only access there is a much smaller blast radius if someone manages to compromise the machine. The problem is that every sing…
Db sharding and replication is a fascinating subject and a matter of deep interest to me. Ben and Matt - appreciate your contributions in this area. I'm interested in making contributions along with you. Please let me know if you are looking for help. Much Thanks.