Database Lab – Full-size staging databases as a service
1–10 of 32 posts
Re: Database Lab – Full-size staging databases as a service
#2This helps a lot in developing and testing for postgres.
Maybe that's one of the tech which "John Carmack" meant when he said sophisticated tech is indistinguishable from magic.
Re: Database Lab – Full-size staging databases as a service
#3This helps a lot in developing and testing for postgres. Maybe that's one of the tech which "John Carmack" meant when he said sophisticated tech is indistinguishable from magic.
John Carmack was not the first to say those words, see [Clarke's law](https://en.wikipedia.org/wiki/Clarke%27s_three_laws)
Re: Database Lab – Full-size staging databases as a service
#4Question: Can this clone a database, but also apply certain operations to it?
I work at a small company, and currently we clone our production database to our dev machines for testing. However certain information in the database is sensitive and we don't want to include it on our dev machines. (This specific sensitive data is also stored in an encrypted format and the key is not included, but we'd still prefer it not included).
Basically I'd like to be able to clone the database, and run some SQL to replace the data in some tables with some mock data. But I can't think of an easy way to do this without cloning the database, starting a temporary one, run the sql, then clone it again - and distribute that final build to the devs.
Re: Database Lab – Full-size staging databases as a service
#5This looks like open source Delphix using Postgres. It even seems to be using ZFS under the hood.
Any comments from people using it? I designed a CI system using Delphix at an old client and it was awesome!
Re: Database Lab – Full-size staging databases as a service
#6This looks like open source Delphix using Postgres. It even seems to be using ZFS under the hood. Any comments from people using it? I designed a CI system using Delphix at an old client and it was awesome!
I saw them contributing often to ZFS on FreeBSD, what is their product? Is it equivalent to NetApp?
Re: Database Lab – Full-size staging databases as a service
#7Question: Can this clone a database, but also apply certain operations to it? I work at a small company, and currently we clone our production database to our dev machines for testing. However certain information in the database is sensitive and we don't want to include it on our dev machines. (This specific sensitive data is also stored in an encrypted format and the key is not included, but we'd still prefer it not…
It doesn't appear to be able to at this point, no. Just clones the existing DB.
There are masking tools out there that can mask data from production in-flight before dropping it into a dev environment for testing (so you can get the same data with the sensitive parts changed), but this doesn't appear to be one of them. I know people who work in the field, it's a tough nut to crack to keep the database good enough to test with while making sure you don't have devs running around with customer data on their laptop/server.
Re: Database Lab – Full-size staging databases as a service
#8Question: Can this clone a database, but also apply certain operations to it? I work at a small company, and currently we clone our production database to our dev machines for testing. However certain information in the database is sensitive and we don't want to include it on our dev machines. (This specific sensitive data is also stored in an encrypted format and the key is not included, but we'd still prefer it not…
It doesn't appear to be able to at this point, no. Just clones the existing DB. There are masking tools out there that can mask data from production in-flight before dropping it into a dev environment for testing (so you can get the same data with the sensitive parts changed), but this doesn't appear to be one of them. I know people who work in the field, it's a tough nut to crack to keep the database good enough to…
> There are masking tools out there that can mask data from production
Thank you so much! "Masking" was the phrase I needed to find some potential solutions! I wasn't able to find much before, that points me in the correct direction. Thanks!
Re: Database Lab – Full-size staging databases as a service
#9This looks like open source Delphix using Postgres. It even seems to be using ZFS under the hood. Any comments from people using it? I designed a CI system using Delphix at an old client and it was awesome!
I saw them contributing often to ZFS on FreeBSD, what is their product? Is it equivalent to NetApp?
Not seen NetApp but from a quick Google it seems to be more intelligent database management.
Delphix is more using the power of ZFS and Copy On Write to give you instant database copies with minimal storage footprint. We used it to give every developer their own Database that they could do anything to and not worry about locks etc. Refreshing to a more recent copy was a very quick operation and required a developer running one command. Had a nice web interface and a REST interface for automation.
I'd recommend it but with the caveat that I've no idea how much it costs as it was a Dev thing a client was encouraging use of. Was really nice to work with though.
Re: Database Lab – Full-size staging databases as a service
#10Question: Can this clone a database, but also apply certain operations to it? I work at a small company, and currently we clone our production database to our dev machines for testing. However certain information in the database is sensitive and we don't want to include it on our dev machines. (This specific sensitive data is also stored in an encrypted format and the key is not included, but we'd still prefer it not…
If you're willing to pay for it Delphix would do what you want. I've not used those features but their tech guys were describing that very scenario in a presentation I saw.