Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Instant database clones with PostgreSQL 18

boringsql.com

Post reply on HN
Print view View original

41–50 of 168 posts

Re: Instant database clones with PostgreSQL 18

#41

Post by foobarbecue »

pebble wrote: post #40
pritambarhate wrote: post #38

Earlier quoted context omitted.

Let's say there is an architect and he also owns a construction company. This architect, then designs a building and gets it built from of his employees and contractors. In such cases the person says, I have built this building. People who found companies, say they have built companies. It's commonly accepted in our society. So even if Claude built for it for GP, as long as GP designed it, paid for tools (Claude) to…

No, it's more like the architect has a cousin who is like "I totally got this bro" and builds the building for them.

Right and also in this world there are no building codes or building inspections.

Re: Instant database clones with PostgreSQL 18

#42

Post by testdelacc1 »

pritambarhate wrote: post #38
teiferer wrote: post #21

Earlier quoted context omitted.

You mean you told Claude a bunch of details and it built it for you? Mind you, I'm not saying it's bad per se. But shouldn't we be open and honest about this? I wonder if this is the new normal. Somebody says "I built Xyz" but then you realize it's vibe coded.

Let's say there is an architect and he also owns a construction company. This architect, then designs a building and gets it built from of his employees and contractors. In such cases the person says, I have built this building. People who found companies, say they have built companies. It's commonly accepted in our society. So even if Claude built for it for GP, as long as GP designed it, paid for tools (Claude) to…

What an outrageously bad analogy. Everyone involved in that building put their professional reputations and licenses on the line. If that building collapses, the people involved will lose their livelihoods and be held criminally liable.

Meanwhile this vibe coded nonsense is provided “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. We don’t even know if he read it before committing and pushing.

Re: Instant database clones with PostgreSQL 18

#43

Post by renewiltord »

dpedu wrote: post #32
teiferer wrote: post #21

Earlier quoted context omitted.

You mean you told Claude a bunch of details and it built it for you? Mind you, I'm not saying it's bad per se. But shouldn't we be open and honest about this? I wonder if this is the new normal. Somebody says "I built Xyz" but then you realize it's vibe coded.

Huh? It says so right in the README. https://github.com/elitan/velo/blame/12712e26b18d0935bfb6c6e... And are we really doing this? Do we need to admit how every line of code was produced? Why? Are you expecting to see "built with the influence of Stackoverflow answers" or "google searches" on every single piece of software ever? It's an exercise of pointlessness.

I think you need to start with the following statement:

> We would like to acknowledge the open source people, who are the traditional custodians of this code. We pay our respects to the stack overflow elders, past, present, and future, who call this place, the code and libraries that $program sits upon, their work. We are proud to continue their tradition of coming together and growing as a community. We thank the search engine for their stewardship and support, and we look forward to strengthening our ties as we continue our relationship of mutual respect and understanding

Then if you would kindly say that a Brazilian invented the airplane that would be good too. If you don’t do this you should be cancelled for your heinous crime.

Re: Instant database clones with PostgreSQL 18

#44

Post by renewiltord »

earthnail wrote: post #25
teiferer wrote: post #21

Earlier quoted context omitted.

You mean you told Claude a bunch of details and it built it for you? Mind you, I'm not saying it's bad per se. But shouldn't we be open and honest about this? I wonder if this is the new normal. Somebody says "I built Xyz" but then you realize it's vibe coded.

Not sure why this is downvoted. For a critical tool like DB cloning, I‘d very much appreciate if it was hand written. Simply because it means it’s also hand reviewed at least once (by definition). We wouldn’t have called it reviewed in the old world, but in the AI coding world we’re now in it makes me realise that yes, it is a form of reviewing. I use Claude a lot btw. But I wouldn’t trust it on mission critical stuf…

If you don’t read code you execute someone is going to steal everything on your file system one day

Re: Instant database clones with PostgreSQL 18

#45

Post by briffle »

proaralyst wrote: post #27
francislavoie wrote: post #23

Is anyone aware of something like this for MariaDB? Something we've been trying to solve for a long time is having instant DB resets between acceptance tests (in CI or locally) back to our known fixture state, but right now it takes decently long (like half a second to a couple seconds, I haven't benchmarked it in a while) and that's by far the slowest thing in our tests. I just want fast snapshotted resets/rewinds t…

You could use LVM or btrfs snapshots (at the filesystem level) if you're ok restarting your database between runs

LVM snapshots work well. Used it for years with other database tools.. But make sure you allocate enough write space for the COW.. when the write space fills up, LVM just 'drops' the snapshot.

Re: Instant database clones with PostgreSQL 18

#46

Post by renewiltord »

francislavoie wrote: post #28
proaralyst wrote: post #27

Earlier quoted context omitted.

You could use LVM or btrfs snapshots (at the filesystem level) if you're ok restarting your database between runs

Restarting the DB is unfortunately way too slow. We run the DB in a docker container with a tmpfs (in-memory) volume which helps a lot with speed, but the problem is still the raw compute needed to wipe the tables and re-fill them with the fixtures every time.

I have not done this so it’s theorycrafting but can’t you do the following?

1. Have a local data dir with initial state

2. Create an overlayfs with a temporary directory

3. Launch your job in your docker container with the overlayfs bind mount as your data directory

4. That’s it. Writes go to the overlay and the base directory is untouched

Re: Instant database clones with PostgreSQL 18

#47

Post by ikatson »

francislavoie wrote: post #28
proaralyst wrote: post #27

Earlier quoted context omitted.

You could use LVM or btrfs snapshots (at the filesystem level) if you're ok restarting your database between runs

Restarting the DB is unfortunately way too slow. We run the DB in a docker container with a tmpfs (in-memory) volume which helps a lot with speed, but the problem is still the raw compute needed to wipe the tables and re-fill them with the fixtures every time.

How about do the changes then bake them into the DB docker image. I.e. "docker commit".

Then spin up the dB using that image instead of an empty one for every test run.

This implies starting the DB through docker is faster than what you're doing now of course.

Re: Instant database clones with PostgreSQL 18

#48

Post by francislavoie »

ikatson wrote: post #47
francislavoie wrote: post #28

Earlier quoted context omitted.

Restarting the DB is unfortunately way too slow. We run the DB in a docker container with a tmpfs (in-memory) volume which helps a lot with speed, but the problem is still the raw compute needed to wipe the tables and re-fill them with the fixtures every time.

How about do the changes then bake them into the DB docker image. I.e. "docker commit". Then spin up the dB using that image instead of an empty one for every test run. This implies starting the DB through docker is faster than what you're doing now of course.

Yeah there's absolutely no way restarting the container will be faster.

Re: Instant database clones with PostgreSQL 18

#49

Post by parthdesai »

notKilgoreTrout wrote: post #31
ffsm8 wrote: post #29

Earlier quoted context omitted.

Eh, DB branching is mostly only necessary for testing - locally, in CI or quick rollbacks on a shared dev instance. Or at least I cannot come up with a usecase for prod. From that perspective, it feels like it'd be a perfect usecase to embrace the LLM guided development jank

Mostly.. App migrations that may fail and need a rollback have the problem that you may not be allowed to wipe any transactions so you may want to be putting data to a parallel world that didn't migrate.

> App migrations that may fail and need a rollback have the problem that you may not be allowed to wipe any transactions so you may want to be putting data to a parallel world that didn't migrate.

This is why migrations are supposed to be backwards compatible

Re: Instant database clones with PostgreSQL 18

#50

Post by francislavoie »

renewiltord wrote: post #46
francislavoie wrote: post #28

Earlier quoted context omitted.

Restarting the DB is unfortunately way too slow. We run the DB in a docker container with a tmpfs (in-memory) volume which helps a lot with speed, but the problem is still the raw compute needed to wipe the tables and re-fill them with the fixtures every time.

I have not done this so it’s theorycrafting but can’t you do the following? 1. Have a local data dir with initial state 2. Create an overlayfs with a temporary directory 3. Launch your job in your docker container with the overlayfs bind mount as your data directory 4. That’s it. Writes go to the overlay and the base directory is untouched

But how does the reset happen fast, the problem isn't with preventing permanent writes or w/e, it's with actually resetting for the next test. Also using overlayfs will immediately be slower at runtime than tmpfs which we're already doing.
Post reply on HN