It looks very interesting for integration tests
would this work inside test containers?
I use CREATE DATABASE dbname TEMPLATE template1; inside test containers. Have not tried this new method yet.
Instant database clones with PostgreSQL 18
141–150 of 168 posts
Re: Instant database clones with PostgreSQL 18
#142Assuming I'd like to replicate my production database for either staging, or to test migrations, etc, and that most of my data is either: - business entities (users, projects, etc) - and "event data" (sent by devices, etc) where most of the database size is in the latter category, and that I'm fine with "subsetting" those (eg getting only the last month's "event data") what would be the best strategy to create a kind…
You can use "psql" to dump subsets of data from tables and then later import them. Something like: psql -c "\copy (SELECT * FROM event_data ORDER BY created_at DESC LIMIT 100) TO 'event-data-sample.csv' WITH CSV HEADER" https://www.postgresql.org/docs/current/sql-copy.html It'd be really nice if pg_dump had a "data sample"/"data subset" option but unfortunately nothing like that is built in that I know of.
Indeed, but is there a way to do it as a "point in time", eg do a "virtual checkpoint" at a timestamp, and do all the copy operations from that timestamp, so they are coherent?
Re: Instant database clones with PostgreSQL 18
#143Earlier quoted context omitted.
I have not done this so it’s theorycrafting but can’t you do the following? 1. Have a local data dir with initial state 2. Create an overlayfs with a temporary directory 3. Launch your job in your docker container with the overlayfs bind mount as your data directory 4. That’s it. Writes go to the overlay and the base directory is untouched
But how does the reset happen fast, the problem isn't with preventing permanent writes or w/e, it's with actually resetting for the next test. Also using overlayfs will immediately be slower at runtime than tmpfs which we're already doing.
Ah I was thinking you just start multiple overlays and run tests independent of each other.
Re: Instant database clones with PostgreSQL 18
#144Earlier quoted context omitted.
You, is an interesting word to use given that you plagiarized it.
Plagiarized from what? Happy to address if you can point to what you're referring to.
I think they may be jumping on the "shit on AI assisted project" bandwagon. I am by no means reaching for ai tools at every turn, but to suggest its plagiarized is laughable.
Don't worry about these trolls.
Re: Instant database clones with PostgreSQL 18
#145Earlier quoted context omitted.
Do you take issue with companies stating that they (the company) built something, instead of stating that their employees built something? Should the architects and senior developers disclaim any credit, because the majority of tickets were completed by junior and mid-level developers? Do you take issue with a CNC machinist stating that they made something, rather than stating that they did the CAD and CAM work but t…
Could we please avoid the strawmen? Nowhere have I claimed that they didn't put work into this. Nowhere did I say that delegation is bad. I'd like to encourage a discussion, but then please counter the opinion that I gave, not a made-up one that I neither stated nor actually hold.
> You mean you told Claude a bunch of details and it built it for you?
> Nowhere have I claimed that they didn't put work into this.
There's some mental gymnastics.
> please counter the opinion that I gave
The reply your responding to did exactly that, and you just gave more snarky responses.
Re: Instant database clones with PostgreSQL 18
#146Earlier quoted context omitted.
Despite all of the complaints in other comments about the use of Claude Code, it looks interesting and I appreciated the video demo you put on the GitHub page.
Agentic coding detractors: "If AI is so great, where all the thriving new open source projects to prove it?" Also agentic coding detractors: "How dare you use AI to help build a new open source project." I'm joking and haven't read the comments you're referring to, but whether or not AI was involved is irrelevant per se. If anyone finds themselves having a gut reaction to "AI", just mentally replace it with "an inter…
For real. For someone to even understand why this tool is useful and functions as intended, they need to have some deeper understanding of software development. Who cares if the implementation was done with AI. With Claude Code, I rarely write code by hand these days, yet my brain hurts more than ever from all the actual problem solving I’m able to drill into with all the programming cruft out of the way. I did it by hand for 15 years, and I don’t feel bad at all for handing that part over.
Re: Instant database clones with PostgreSQL 18
#147Earlier quoted context omitted.
> App migrations that may fail and need a rollback have the problem that you may not be allowed to wipe any transactions so you may want to be putting data to a parallel world that didn't migrate. This is why migrations are supposed to be backwards compatible
https://github.com/flyway/flywaydb.org/blob/gh-pages/documen... You can certainly bet you followed that advice correctly, now what are the odds you could test a what-if like that in sufficient depth?
[deleted]
Re: Instant database clones with PostgreSQL 18
#148PostgreSQL seems to have become the be-all, end-all SQL database that does everything and does it all well. And it's free! I'm wondering why anyone would want to use anything else at this point (for SQL).
To be fair, postgres still suffers from a poor choice of MVCC implementation (copy on write rather than an undo log). This one small choice has a huge number of negative knock on effects once your load becomes non-trivial
Re: Instant database clones with PostgreSQL 18
#149Earlier quoted context omitted.
Agentic coding detractors: "If AI is so great, where all the thriving new open source projects to prove it?" Also agentic coding detractors: "How dare you use AI to help build a new open source project." I'm joking and haven't read the comments you're referring to, but whether or not AI was involved is irrelevant per se. If anyone finds themselves having a gut reaction to "AI", just mentally replace it with "an inter…
For real. For someone to even understand why this tool is useful and functions as intended, they need to have some deeper understanding of software development. Who cares if the implementation was done with AI. With Claude Code, I rarely write code by hand these days, yet my brain hurts more than ever from all the actual problem solving I’m able to drill into with all the programming cruft out of the way. I did it by…
A decade ago, a senior staff engineer at Google told me that he doesn't mind delegating the data-entry parts of his job to junior SWEs, so he can focus on higher-level problem solving.
This is how I've been treating AI, except instead of assuming your junior SWE is generally sane and has some understand of what you're doing, you have to make sure you double check everything.
Re: Instant database clones with PostgreSQL 18
#150I set this up for my employer many years ago when they migrated to RDS. We kept bumping into issues on production migrations that would wreck things. I decided to do something about it. The steps were basically: 1. Clone the AWS RDS db - or spin up a new instance from a fresh backup. 2. Get the arn and from that the cname or public IP. 3. Plug that into the DB connection in your app 4. Run the migration on pseudo pro…
I love those "migration only fails in prod because of data quirks" bugs. They are the freaking worst. Have called off releases in the past because of it.
You should almost never test in prod, but sometimes testing on [a copy of] prod is useful