Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Database Lab – Full-size staging databases as a service

gitlab.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Database Lab – Full-size staging databases as a service

#21

Post by samokhvalov »

gavinray wrote: post #18
samokhvalov wrote: post #17

Earlier quoted context omitted.

Well, not really. Two key differences: 1) dump/restore approach is one of "thick cloning" methods. It will take a really significant amount of time. Roughly, thick cloning speed for modern hardware and networks is somewhat ~1 TiB / hour just for transferring over network and writing to disk. Additionally, in the case of dump/restore, the timing will depend on the number of indexes that need to be built, materialized…

Ahh okay, that makes sense. I wasn't aware that using different methods of copying + backing up data had tangible effects on it later. Super interesting, and vital if you're looking to gather accurate data from experiments. By the way, I recognized you guys from the other tool you publish, Nancy. While I don't have a direct usecase for it myself, I thought the idea was super valuable as well. I am a huge Postgres guy…

Thank you! I'll check out aquameta, looks interesting.

Re: Database Lab – Full-size staging databases as a service

#22

Post by sandGorgon »

My advice - go all the way into RDS territory.

This is a killer feature. But it is hard to adopt it by itself, while being on RDS, which is what most people end up doing.

If you can build a management layer that gives better-than-RDS features ...while allowing me to BYOB my own cloud, you are on your way to win.

Re: Database Lab – Full-size staging databases as a service

#23

Post by samokhvalov »

hdhdudidd wrote: post #19
samokhvalov wrote: post #17

Earlier quoted context omitted.

Well, not really. Two key differences: 1) dump/restore approach is one of "thick cloning" methods. It will take a really significant amount of time. Roughly, thick cloning speed for modern hardware and networks is somewhat ~1 TiB / hour just for transferring over network and writing to disk. Additionally, in the case of dump/restore, the timing will depend on the number of indexes that need to be built, materialized…

Can you explain what a thin clone is?

Sure. Thin cloning is a way to obtain a clone of data that looks fully independent (you can modify, add, delete data), but physically multiple clones share physically the same data -- except those blocks that were changed. This is implemented using copy-on-write -- see, for example, https://en.wikipedia.org/wiki/Copy-on-write#In_computer_stor...

Re: Database Lab – Full-size staging databases as a service

#24

Post by jlmorton »

sandGorgon wrote: post #22

My advice - go all the way into RDS territory. This is a killer feature. But it is hard to adopt it by itself, while being on RDS, which is what most people end up doing. If you can build a management layer that gives better-than-RDS features ...while allowing me to BYOB my own cloud, you are on your way to win.

I don't think that's necessary, and it might even detract a bit.

At my last company, we built a very similar tool in-house, using LVM w/ thin snapshots, instead of ZFS. The database snapshot host would receive streaming logical replication from our UAT and Staging environments.

Users (developers, QA, etc) could then take an instant snapshot of a multi-TB database, provide a Git hash/branch, and we'd instantly clone the DB, spin up the application and migrate it to the latest version.

That worked great, and the same thing could work with RDS. You could use something like AWS Database Migration Service to replicate from your RDS installation to your Database Lab host.

This way, you get to keep RDS for your actual test and staging environments, but you get streaming replication into your Database Manager instance for Dev/QA.

Re: Database Lab – Full-size staging databases as a service

#27

Post by sandGorgon »

jlmorton wrote: post #24
sandGorgon wrote: post #22

My advice - go all the way into RDS territory. This is a killer feature. But it is hard to adopt it by itself, while being on RDS, which is what most people end up doing. If you can build a management layer that gives better-than-RDS features ...while allowing me to BYOB my own cloud, you are on your way to win.

I don't think that's necessary, and it might even detract a bit. At my last company, we built a very similar tool in-house, using LVM w/ thin snapshots, instead of ZFS. The database snapshot host would receive streaming logical replication from our UAT and Staging environments. Users (developers, QA, etc) could then take an instant snapshot of a multi-TB database, provide a Git hash/branch, and we'd instantly clone t…

> You could use something like AWS Database Migration Service to replicate from your RDS installation to your Database Lab host.

my guess is that then your market will be restricted to those who are already running their own databases. Because RDS restore-from-snapshot is good enough to not go through the DMS hackery (which is exceedingly nasty) and probably will negate some of the "instant" features that you have.

Just my $0.02 though.

Re: Database Lab – Full-size staging databases as a service

#28

Post by francislavoie »

Does such a thing exist for MySQL? If so I'd appreciate a link. We have weekly snapshots that we use for development, but it would be amazing to have a way to reset/throw away any changes quickly, especially to repeatedly test a specific use-case against some prod data.

Re: Database Lab – Full-size staging databases as a service

#30

Post by purrpit »

geoffbp wrote: post #29

What are people using these days for data sanitization? I.e. to remove PII data from dev snapshots

Short answer: Different databases.

We have PII, analytics, catalog, transaction etc databases. We only let people clone non-sensitive databases. Others are cloned only at schema level and data is left out.

This also allows us to move and scale independently across different applications. But that's just a symptom of this design.

There are some complications with this approach but it's a trade off and we try to come on top using foreign-data-wrapper approach and using a database application router for read and writes. We have open sourced a Django db router which routes queries on the basis of tags/app name/table name etc

Post reply on HN