Show HN: EDB – A framework to make and manage backups of your database
1–9 of 9 posts
Re: Show HN: EDB – A framework to make and manage backups of your database
#2Filed issues related to crypto: https://github.com/RoxasShadow/EDB/issues/created_by/dchest
Re: Show HN: EDB – A framework to make and manage backups of your database
#3Simple, clean, straightforward. I've done similar things in bash before but not bothered to publish :) It's a good idea though to have an ecosystem of simple but flexible tools like this.
Here's another idea we implemented alongside a backup system like this. Say you run a backup daily; you can run another script to prune the archives, keeping D most recent daily archives, W most recent weekly archives, M most recent monthly archives, and all yearly archives.
Re: Show HN: EDB – A framework to make and manage backups of your database
#4Simple, clean, straightforward. I've done similar things in bash before but not bothered to publish :) It's a good idea though to have an ecosystem of simple but flexible tools like this. Here's another idea we implemented alongside a backup system like this. Say you run a backup daily; you can run another script to prune the archives, keeping D most recent daily archives, W most recent weekly archives, M most recent…
Thank you for your feedback.
That would suite well as module when used alongside the FTP module. I'll keep it in mind :)
Re: Show HN: EDB – A framework to make and manage backups of your database
#5Filed issues related to crypto: https://github.com/RoxasShadow/EDB/issues/created_by/dchest
Thanks for the good points raised!
Re: Show HN: EDB – A framework to make and manage backups of your database
#6I've recently created a similar tool at our company.
We found out that dumping the database table by table in a parallelized way is much faster than a full database mysqldump.
There's also mysqlpump soon available in mysql 5.7 to replace mysqldump and works in parallel.
Re: Show HN: EDB – A framework to make and manage backups of your database
#7The mysql driver needs additional work. In particular, it needs a '--single-transaction' flag, or a global lock, to ensure that the dump is consistent - particularly if you want to dump multiple databases concurrently.
Doing one dump per table with chunking (each file has N rows) would help with both speed and disk sizes of backups by allowing S3 or some other program to implement de-duplication between incremental backups.
It also wouldn't hurt to capture the binlog position, if available, to enable point in time recovery.
Have a look at mydumper for an idea of how another tool implemented these:
Re: Show HN: EDB – A framework to make and manage backups of your database
#8I've recently created a similar tool at our company. We found out that dumping the database table by table in a parallelized way is much faster than a full database mysqldump. There's also mysqlpump soon available in mysql 5.7 to replace mysqldump and works in parallel.
Until then, have a look at "mydumper", which does both parallel dumps at a table level, but also chunking, which makes de-duplication possible.
Re: Show HN: EDB – A framework to make and manage backups of your database
#9The mysql driver needs additional work. In particular, it needs a '--single-transaction' flag, or a global lock, to ensure that the dump is consistent - particularly if you want to dump multiple databases concurrently. Doing one dump per table with chunking (each file has N rows) would help with both speed and disk sizes of backups by allowing S3 or some other program to implement de-duplication between incremental b…
I will open an issue with your suggestion. Thanks.