Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean lost our data and gave us $500

dfernandez.me

Post reply on HN
Print view View original

61–69 of 69 posts

Re: DigitalOcean lost our data and gave us $500

#61

Post by emeraldd »

sillysaurus2 wrote: post #31
wiredfool wrote: post #21

Earlier quoted context omitted.

As long as you issue the pg_start_backup/pg_stop_backup pair and keep the WAL logs. If you don't, then you've got a corrupt backup. At least you would catch that problem in your first test restore.

... what? The point of consistency is that if the power to your server is cut, then you can reboot and pick up precisely where you left off. That means the database on disk must have consistency. Meaning you should be able to copy it at any point in time without any problems. If you can't, then that's not consistency, and if postgres really works that way, then it's failing one of the basic tenants of being a databas…

There's something that works and there's the right way to do it. It's better to do things the right way if you want to make sure everything is in a good state when you bring it back and there aren't edge cases you missed ... What if someone forgot to use a transaction?

Re: DigitalOcean lost our data and gave us $500

#62

Post by neom »

I truly believe that we did the best we could in this instance. Drive failures are always always unfortunate, even with backups, downtime exists.

That being said, we're always genuinely looking to improve, and I'd welcome your feedback on how you feel we did and how you feel we could do better. Please do reach out to me personally john@do! Thanks. :)

Re: DigitalOcean lost our data and gave us $500

#63

Post by AznHisoka »

jtchang wrote: post #25
AznHisoka wrote: post #9

So this is a technical problem I am having right now that's preventing me from backing up a Postgres database completely (hope someone here can help). I have a master Postgres database that is receiving a TON of transactions per second (I'm talking about thousand concurrent transactions). We tried running pg_dump on this database, but the DB is just too huge, and it took more than 4 days to completely dump out everyt…

I use postgres and ran into this issue as well. Inside postgres.conf for the slave I have the following: # These settings are ignored on a master server. hot_standby = on # "on" allows queries during recovery # (change requires restart) max_standby_archive_delay = 900s # max delay before canceling queries # when reading WAL from archive; # -1 allows indefinite delay max_standby_streaming_delay = 900s # max delay befo…

So you basically sacrifice speed of replication in order to ensure long running queries don't get cancelled?

Re: DigitalOcean lost our data and gave us $500

#64

Post by AznHisoka »

wiredfool wrote: post #21
sillysaurus2 wrote: post #16

Earlier quoted context omitted.

Was the database designed using transactions to achieve consistency? If so, then you can just instruct Tarsnap to back up the folder containing your database every day, and you're done. If the DB uses transactions for consistency, you can copy it at any time without any problems.

As long as you issue the pg_start_backup/pg_stop_backup pair and keep the WAL logs. If you don't, then you've got a corrupt backup. At least you would catch that problem in your first test restore.

Does this interfere with the replication process at all if I run pg_start and pg_stop_backup, and rsync the files to another server?

Re: DigitalOcean lost our data and gave us $500

#65

Post by hga »

gregd wrote: post #18

This is 2013. Why are we still talking about backups as a lesson learned? Is it because startups are skimping on Sys Admins?

Do you know of anyone who didn't get deadly serious about backups before they had a sour taste of data loss?

Me, I was just lucky my first really interesting experience was on a big UNIX(TM) Version 6 system, with a couple of user accessible DECTapes. Buying a tape was cheap enough, and the whole thing was neat ... and then I learned the -rf flags to rm. And had any critical data I lost on that DECTape.

Today I do nightly backups of my home systems to LTO-4 tapes (as well as offsite of the most critical to rsync.net a time zone away).

Re: DigitalOcean lost our data and gave us $500

#67

Post by wiredfool »

AznHisoka wrote: post #64
wiredfool wrote: post #21

Earlier quoted context omitted.

As long as you issue the pg_start_backup/pg_stop_backup pair and keep the WAL logs. If you don't, then you've got a corrupt backup. At least you would catch that problem in your first test restore.

Does this interfere with the replication process at all if I run pg_start and pg_stop_backup, and rsync the files to another server?

Nope. I do it all the time.

The start/stop backup has to be issued on the master. It doesn't look like the standby gets the backup label (at least on 9.0, may have changed since). So you'd have to be reading from the master's data directory.

Alternately, you could stop the secondary and pull from there. But that interrupts the replication, and then the secondary would have to catch up, which might be hard depending on your level of usage.

feel free to email or chat my un on freenode.

Re: DigitalOcean lost our data and gave us $500

#68

Post by wiredfool »

AznHisoka wrote: post #63
jtchang wrote: post #25

Earlier quoted context omitted.

I use postgres and ran into this issue as well. Inside postgres.conf for the slave I have the following: # These settings are ignored on a master server. hot_standby = on # "on" allows queries during recovery # (change requires restart) max_standby_archive_delay = 900s # max delay before canceling queries # when reading WAL from archive; # -1 allows indefinite delay max_standby_streaming_delay = 900s # max delay befo…

So you basically sacrifice speed of replication in order to ensure long running queries don't get cancelled?

It's sacrificing the expected latency of replication.

Incidentally, if you're on 9.3 and your HW can handle it, take a look at parallelizing the pg_dump. If you've got a relatively fast disk subsystem and many cores, you can get a speedup. I've found it tends to make the dumps O(biggest table) instead of O(sum of all tables).

(It's native on 9.3, I've hacked up some scripts that do it for 9.0, but they don't get a consistent snapshot, so I do it during scheduled downtime. OTOH, the dump/restore is ~6x faster OMM/OMD, so the downtime is that much shorter)

Re: DigitalOcean lost our data and gave us $500

#69

Post by tovmeod »

asiekierka wrote: post #7
sillysaurus2 wrote: post #4

So if you were backing up your data to Tarsnap, then you'd be up and running as quickly as you could launch a new instance and redownload everything. And $500 credit is enough to power a micro droplet for 100 months, or a small droplet for 50 months. DO handled this well. http://www.tarsnap.com EDIT: s/years/months/g. Thanks.

You mean months.

50 months == 4 years and 2 months 100 months == 8 years and 4 months
Post reply on HN