Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What if database branching was easy?

xata.io

Post reply on HN
Print view View original

21–30 of 59 posts

Re: What if database branching was easy?

#21

Post by tee-es-gee »

sastraxi wrote: post #5

I’ve done experiments using BTRFS and ZFS for local Postgres copy-on-write. You don’t need anything but vanilla pg and a supported file system to do it anymore; just clone the database using a template and a newish version of Postgres. Looking at Xata’s technical deep dive, the site claims that we need an additional Postgres instance per replica and proposes a network file system to work around that. But I don’t real…

For context for the others, I think you are referring to this blog post: https://xata.io/blog/open-source-postgres-branching-copy-on-... (in particular the "The key is in the storage system" section) right?

What I'm saying there is that if you do Postgres with on top of a local ZFS volume, the child branches Postgres instances need to be on the same server. So you are limited in how many branches you can do. One or two are fine, but if you want to do a branch per PR, that will likely not work.

If you separate the compute from storage via the network, this problem goes away.

Re: What if database branching was easy?

#22

Post by xeubie »

camdez wrote: post #19
xeubie wrote: post #9

Earlier quoted context omitted.

Surprisingly, neither Datomic nor XTDB support branching. See: https://blog.danieljanus.pl/datomic-forking-the-past/ I actually built my own immutable database which does support branching (see profile), so it seems like a huge miss that these ones don't. It's pretty much the main reason I would want an immutable database.

Wait, this statement seems way too strong. The linked article points out that Datomic doesn't support branching from the past . It absolutely does support branching, and I've built entire test suites that way. From a cursory glance, I'd say Datomic does exactly what the original parent article is discussing. It works great and it's super convenient.

If each "branch" is read only, it's not a branch at all. The entire idea of branching implies that you can make changes on one branch, then switch to another branch and make changes to it. They start from the same point and grow in different directions, as the metaphor of branches on a tree depicts.

Re: What if database branching was easy?

#23

Post by wadefletch »

sastraxi wrote: post #5

I’ve done experiments using BTRFS and ZFS for local Postgres copy-on-write. You don’t need anything but vanilla pg and a supported file system to do it anymore; just clone the database using a template and a newish version of Postgres. Looking at Xata’s technical deep dive, the site claims that we need an additional Postgres instance per replica and proposes a network file system to work around that. But I don’t real…

You can't have any other connections while a Copy-on-Write is happening, not even a logical replication slot. So you keep a read replica that then gets all connections briefly cut for the COW to avoid locking the master instance. Then you re-enable the logical replication slots on both the new, copied instance and the "copyable" read replica to get both back up to date w/ master.

Re: What if database branching was easy?

#25

Post by zwily »

miketery wrote: post #10
theaniketmaurya wrote: post #4

i was using neon and they had some similar feature but now using planetscale. would be curious to know how you all are doing it?

We used neon at last job. It seemed pretty cool. What made you switch to planetscale?

Planetscale has better performance and uptime. But the branching keeps me going back to neon…

Yes planetscale can branch too, but it takes longer and you pay individually for each branch

Re: What if database branching was easy?

#26

Post by tee-es-gee »

sastraxi wrote: post #5

I’ve done experiments using BTRFS and ZFS for local Postgres copy-on-write. You don’t need anything but vanilla pg and a supported file system to do it anymore; just clone the database using a template and a newish version of Postgres. Looking at Xata’s technical deep dive, the site claims that we need an additional Postgres instance per replica and proposes a network file system to work around that. But I don’t real…

> You don’t need anything but vanilla pg and a supported file system to do it anymore; just clone the database using a template and a newish version of Postgres.

Are you referring to `file_copy_method = clone` from Postgres 18? For example: https://boringsql.com/posts/instant-database-clones/

I think the key limitation is:

> The source database can't have any active connections during cloning. This is a PostgreSQL limitation, not a filesystem one.

Re: What if database branching was easy?

#29

Post by gulcin »

zwily wrote: post #25
miketery wrote: post #10

Earlier quoted context omitted.

We used neon at last job. It seemed pretty cool. What made you switch to planetscale?

Planetscale has better performance and uptime. But the branching keeps me going back to neon… Yes planetscale can branch too, but it takes longer and you pay individually for each branch

Xata is open-source now, maybe you can give it a try as an alternative to Neon.
Post reply on HN