Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Instant database clones with PostgreSQL 18

boringsql.com

Post reply on HN
Print view View original

21–30 of 168 posts

Re: Instant database clones with PostgreSQL 18

#21

Post by teiferer »

elitan wrote: post #17

For those who can't wait for PG18 or need full instance isolation: I built Velo, which does instant branching using ZFS snapshots instead of reflinks. Works with any PG version today. Each branch is a fully isolated PostgreSQL container with its own port. ~2-5 seconds for a 100GB database. https://github.com/elitan/velo Main difference from PG18's approach: you get complete server isolation (useful for testing migrat…

You mean you told Claude a bunch of details and it built it for you?

Mind you, I'm not saying it's bad per se. But shouldn't we be open and honest about this?

I wonder if this is the new normal. Somebody says "I built Xyz" but then you realize it's vibe coded.

Re: Instant database clones with PostgreSQL 18

#22

Post by oulipo2 »

Assuming I'd like to replicate my production database for either staging, or to test migrations, etc,

and that most of my data is either:

- business entities (users, projects, etc)

- and "event data" (sent by devices, etc)

where most of the database size is in the latter category, and that I'm fine with "subsetting" those (eg getting only the last month's "event data")

what would be the best strategy to create a kind of "staging clone"? ideally I'd like to tell the database (logically, without locking it expressly): do as though my next operations only apply to items created/updated BEFORE "currentTimestamp", and then:

- copy all my business tables (any update to those after currentTimestamp would be ignored magically even if they happen during the copy) - copy a subset of my event data (same constraint)

what's the best way to do this?

Re: Instant database clones with PostgreSQL 18

#23

Post by francislavoie »

Is anyone aware of something like this for MariaDB?

Something we've been trying to solve for a long time is having instant DB resets between acceptance tests (in CI or locally) back to our known fixture state, but right now it takes decently long (like half a second to a couple seconds, I haven't benchmarked it in a while) and that's by far the slowest thing in our tests.

I just want fast snapshotted resets/rewinds to a known DB state, but I need to be using MariaDB since it's what we use in production, we can't switch DB tech at this stage of the project, even though Postgres' grass looks greener.

Re: Instant database clones with PostgreSQL 18

#24

Post by whalesalad »

elitan wrote: post #17

For those who can't wait for PG18 or need full instance isolation: I built Velo, which does instant branching using ZFS snapshots instead of reflinks. Works with any PG version today. Each branch is a fully isolated PostgreSQL container with its own port. ~2-5 seconds for a 100GB database. https://github.com/elitan/velo Main difference from PG18's approach: you get complete server isolation (useful for testing migrat…

Hell yeah. I’ve been meaning to prototype this exact thing but with btrfs.

Re: Instant database clones with PostgreSQL 18

#25

Post by earthnail »

teiferer wrote: post #21
elitan wrote: post #17

For those who can't wait for PG18 or need full instance isolation: I built Velo, which does instant branching using ZFS snapshots instead of reflinks. Works with any PG version today. Each branch is a fully isolated PostgreSQL container with its own port. ~2-5 seconds for a 100GB database. https://github.com/elitan/velo Main difference from PG18's approach: you get complete server isolation (useful for testing migrat…

You mean you told Claude a bunch of details and it built it for you? Mind you, I'm not saying it's bad per se. But shouldn't we be open and honest about this? I wonder if this is the new normal. Somebody says "I built Xyz" but then you realize it's vibe coded.

Not sure why this is downvoted. For a critical tool like DB cloning, I‘d very much appreciate if it was hand written. Simply because it means it’s also hand reviewed at least once (by definition).

We wouldn’t have called it reviewed in the old world, but in the AI coding world we’re now in it makes me realise that yes, it is a form of reviewing.

I use Claude a lot btw. But I wouldn’t trust it on mission critical stuff.

Re: Instant database clones with PostgreSQL 18

#26

Post by Rovanion »

elitan wrote: post #17

For those who can't wait for PG18 or need full instance isolation: I built Velo, which does instant branching using ZFS snapshots instead of reflinks. Works with any PG version today. Each branch is a fully isolated PostgreSQL container with its own port. ~2-5 seconds for a 100GB database. https://github.com/elitan/velo Main difference from PG18's approach: you get complete server isolation (useful for testing migrat…

You, is an interesting word to use given that you plagiarized it.

Re: Instant database clones with PostgreSQL 18

#27

Post by proaralyst »

francislavoie wrote: post #23

Is anyone aware of something like this for MariaDB? Something we've been trying to solve for a long time is having instant DB resets between acceptance tests (in CI or locally) back to our known fixture state, but right now it takes decently long (like half a second to a couple seconds, I haven't benchmarked it in a while) and that's by far the slowest thing in our tests. I just want fast snapshotted resets/rewinds t…

You could use LVM or btrfs snapshots (at the filesystem level) if you're ok restarting your database between runs

Re: Instant database clones with PostgreSQL 18

#28

Post by francislavoie »

proaralyst wrote: post #27
francislavoie wrote: post #23

Is anyone aware of something like this for MariaDB? Something we've been trying to solve for a long time is having instant DB resets between acceptance tests (in CI or locally) back to our known fixture state, but right now it takes decently long (like half a second to a couple seconds, I haven't benchmarked it in a while) and that's by far the slowest thing in our tests. I just want fast snapshotted resets/rewinds t…

You could use LVM or btrfs snapshots (at the filesystem level) if you're ok restarting your database between runs

Restarting the DB is unfortunately way too slow. We run the DB in a docker container with a tmpfs (in-memory) volume which helps a lot with speed, but the problem is still the raw compute needed to wipe the tables and re-fill them with the fixtures every time.

Re: Instant database clones with PostgreSQL 18

#29

Post by ffsm8 »

earthnail wrote: post #25
teiferer wrote: post #21

Earlier quoted context omitted.

You mean you told Claude a bunch of details and it built it for you? Mind you, I'm not saying it's bad per se. But shouldn't we be open and honest about this? I wonder if this is the new normal. Somebody says "I built Xyz" but then you realize it's vibe coded.

Not sure why this is downvoted. For a critical tool like DB cloning, I‘d very much appreciate if it was hand written. Simply because it means it’s also hand reviewed at least once (by definition). We wouldn’t have called it reviewed in the old world, but in the AI coding world we’re now in it makes me realise that yes, it is a form of reviewing. I use Claude a lot btw. But I wouldn’t trust it on mission critical stuf…

Eh, DB branching is mostly only necessary for testing - locally, in CI or quick rollbacks on a shared dev instance.

Or at least I cannot come up with a usecase for prod.

From that perspective, it feels like it'd be a perfect usecase to embrace the LLM guided development jank

Re: Instant database clones with PostgreSQL 18

#30

Post by anonymars »

Rovanion wrote: post #26
elitan wrote: post #17

For those who can't wait for PG18 or need full instance isolation: I built Velo, which does instant branching using ZFS snapshots instead of reflinks. Works with any PG version today. Each branch is a fully isolated PostgreSQL container with its own port. ~2-5 seconds for a 100GB database. https://github.com/elitan/velo Main difference from PG18's approach: you get complete server isolation (useful for testing migrat…

You, is an interesting word to use given that you plagiarized it.

Do you have a link to the original?
Post reply on HN