Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean lost our data and gave us $500

dfernandez.me

Post reply on HN
Print view View original

21–30 of 69 posts

Re: DigitalOcean lost our data and gave us $500

#21

Post by wiredfool »

sillysaurus2 wrote: post #16
AznHisoka wrote: post #9

So this is a technical problem I am having right now that's preventing me from backing up a Postgres database completely (hope someone here can help). I have a master Postgres database that is receiving a TON of transactions per second (I'm talking about thousand concurrent transactions). We tried running pg_dump on this database, but the DB is just too huge, and it took more than 4 days to completely dump out everyt…

Was the database designed using transactions to achieve consistency? If so, then you can just instruct Tarsnap to back up the folder containing your database every day, and you're done. If the DB uses transactions for consistency, you can copy it at any time without any problems.

As long as you issue the pg_start_backup/pg_stop_backup pair and keep the WAL logs. If you don't, then you've got a corrupt backup.

At least you would catch that problem in your first test restore.

Re: DigitalOcean lost our data and gave us $500

#22

Post by [deleted] »

AznHisoka wrote: post #9

So this is a technical problem I am having right now that's preventing me from backing up a Postgres database completely (hope someone here can help). I have a master Postgres database that is receiving a TON of transactions per second (I'm talking about thousand concurrent transactions). We tried running pg_dump on this database, but the DB is just too huge, and it took more than 4 days to completely dump out everyt…

[deleted]

Re: DigitalOcean lost our data and gave us $500

#23

Post by mikeocool »

AznHisoka wrote: post #9

So this is a technical problem I am having right now that's preventing me from backing up a Postgres database completely (hope someone here can help). I have a master Postgres database that is receiving a TON of transactions per second (I'm talking about thousand concurrent transactions). We tried running pg_dump on this database, but the DB is just too huge, and it took more than 4 days to completely dump out everyt…

pg_dump is a logical backup, that is, as you've seen, it queries the all of the data in your database in writes to a file in the form of queries that will re-create all of your data in a new database. This great and very flexible, but as you've seen has some limitations.

You probably want to look into physical backups, where you basically copy the actual files that postgres is actually using to store your databases on disk (although it's not quite that simple, so do some googling on it). This has the nice advantage of not requiring you to run queries against your database to back it up. It also gives you a consistent point-in-time backup of your database.

Re: DigitalOcean lost our data and gave us $500

#24

Post by viraptor »

> And if you just launched and have a single instance running, let your alpha users know that there will probably be some downtime.

That's true. But there's no reason for extended downtime even if that instance goes down. Make sure your whole setup is described in chef/puppet/salt/ansible/cf/whatever and even a rebuild from scratch takes only minutes then. There's really little reason to skip that these days.

Re: DigitalOcean lost our data and gave us $500

#25

Post by jtchang »

AznHisoka wrote: post #9

So this is a technical problem I am having right now that's preventing me from backing up a Postgres database completely (hope someone here can help). I have a master Postgres database that is receiving a TON of transactions per second (I'm talking about thousand concurrent transactions). We tried running pg_dump on this database, but the DB is just too huge, and it took more than 4 days to completely dump out everyt…

I use postgres and ran into this issue as well.

Inside postgres.conf for the slave I have the following:

# These settings are ignored on a master server.

hot_standby = on # "on" allows queries during recovery # (change requires restart) max_standby_archive_delay = 900s # max delay before canceling queries # when reading WAL from archive; # -1 allows indefinite delay max_standby_streaming_delay = 900s # max delay before canceling queries # when reading streaming WAL; # -1 allows indefinite delay #wal_receiver_status_interval = 10s # send replies at least this often # 0 disables #hot_standby_feedback = off # send info from standby to prevent # query conflicts

So I set it to 15 minutes for this specific backup server which I am okay with. I already have another server with much shorter time delays.

Re: DigitalOcean lost our data and gave us $500

#26

Post by toomuchtodo »

Liongadev wrote: post #20

What is the best/cost effectiv way to backup a windows server?

Backup the data and configuration information to an object store (AWS S3), use configuration management tools so you can programmatically provision a new server (dedicated or virtual, doesn't matter) in the event of failure. Provisioning should include functionality to deploy your application, and to restore your data to whatever data storage application (SQL, NoSQL, etc) you're using.

If you have questions, more than happy you provide free advice.

Re: DigitalOcean lost our data and gave us $500

#27

Post by FooBarWidget »

AznHisoka wrote: post #9

So this is a technical problem I am having right now that's preventing me from backing up a Postgres database completely (hope someone here can help). I have a master Postgres database that is receiving a TON of transactions per second (I'm talking about thousand concurrent transactions). We tried running pg_dump on this database, but the DB is just too huge, and it took more than 4 days to completely dump out everyt…

Sounds to me like you have no choice but to shard your database in order to reduce the write load on a single database.

Or upgrade your database servers' hardware (more RAM, faster CPU, faster SSD) to the point where you can dump the database.

Re: DigitalOcean lost our data and gave us $500

#28

Post by jtchang »

AznHisoka wrote: post #9

So this is a technical problem I am having right now that's preventing me from backing up a Postgres database completely (hope someone here can help). I have a master Postgres database that is receiving a TON of transactions per second (I'm talking about thousand concurrent transactions). We tried running pg_dump on this database, but the DB is just too huge, and it took more than 4 days to completely dump out everyt…

By the way if you need help feel free to e-mail/ping me.
Post reply on HN