Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

'Check Your Backups Work' Day

checkyourbackups.work

Post reply on HN
Print view View original

81–90 of 152 posts

Re: 'Check Your Backups Work' Day

#81

Post by cuu508 »

dspillett wrote: post #45
soheil wrote: post #21

Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…

> even then it could still just stop working one day This is where automated testing helps. My mail server has a sister VM out in the wild that once a day picks up the latest backup from the offsite-online backups and restores it, sending me a copy of the last message received according to the data in that backup. If I don't get the message, restoring the backup failed. If the message looks too old then making or tra…

> picks up the latest backup from the offsite-online backups and restores it, sending me a copy of the last message received according to the data in that backup.

This is sensible, but one problem with this is developing "blindness" for messages you receive every day. For some recurring tasks you can automate a bit further. Instead of receiving a success message every day, receive a failure message the first day there's an absence of success message. A number of tools and services exist for setting this up, one is linked in my HN profile (shameless plug ;-)

Re: 'Check Your Backups Work' Day

#82

Post by sytse »

ocdtrekkie wrote: post #56
tawayway wrote: post #32

Earlier quoted context omitted.

> Imagine if you're the CEO of Gitlab and seeing this right now I kind of hope the CEO of Gitlab isn't reading HN right now

If he's getting some much needed sleep right now, he'll probably read it tomorrow. The CEO of GitLab is really active on HN, and a pretty classy guy.

Thanks. Just landed in Europe for Git Merge. It is heartwarming to read some of the comments here. We realize that we only get this leniency once and we'll learn from this and communicate our lessons in a few days from now.

Re: 'Check Your Backups Work' Day

#83

Post by jivid »

elvinyung wrote: post #6
HarrisonFisk wrote: post #5

This is every day at Facebook! https://code.facebook.com/posts/1007323976059780/continuous-...

Oh wow, some interesting stuff in here. Looks like they use MySQL as a queue for scheduling the ORC Peons? Would have loved to hear more about why they did that.

Author here.

We use it because it works well for us. We've put a lot of work into making MySQL scale for us to the point where it's a very well supported system and one of the main choices for a lot of storage decisions.

We even use MySQL as a queue for Facebook Messenger. More details about this:

https://www.youtube.com/watch?v=eADBCKKf8PA

https://code.facebook.com/posts/820258981365363/building-mob...

Re: 'Check Your Backups Work' Day

#84

Post by djm_ »

sytse wrote: post #67
edelans wrote: post #62

Earlier quoted context omitted.

Agreed ! Even when they screw things up, they make people benefit from it! Kudos for such a good spirit and dedication to your customers. Everybody screw up from time to time in our industry. And when this happens, you have 2 types of guy : those who try to hide it, and those like Giltlab team who communicate as fast as they can because they respect their customers. Paradoxically, to me, it creates more trust than it…

Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and…

Just know that your transparency here in this situation has put you leagues and bounds ahead of other vendors in my mind.

Thank you for sharing.

Re: 'Check Your Backups Work' Day

#85

Post by elvinyung »

hueving wrote: post #42
elvinyung wrote: post #6

Earlier quoted context omitted.

Oh wow, some interesting stuff in here. Looks like they use MySQL as a queue for scheduling the ORC Peons? Would have loved to hear more about why they did that.

Because it works perfectly fine? I tend to write anyone off who scoffs at simple database-as-queue designs without understanding what the scaling requirements are. You can use a database as a job queue for 10s of thousands of jobs per day without any sweat.

Please forgive me, but I don't understand your hostility.

I understand that there are perfectly legitimate reasons for using a database as a queue. If you frequently need to look at and rearrange your jobs while they're in flight, chances are a pure FIFO structure probably doesn't work that well for you anyways. If the enqueue is contingent on a transaction committing, probably makes a lot of sense for the job to be in the same database. You don't need to tell me -- I've seen more than a few in production.

But an actual message queue also "works perfectly fine" based on the information provided, and I would imagine that a company like Facebook probably already has a few of those lying around. It would have been a conscious choice to use MySQL as a queue.

I swear I'm actually, seriously just curious!

Re: 'Check Your Backups Work' Day

#86

Post by elvinyung »

jivid wrote: post #83
elvinyung wrote: post #6

Earlier quoted context omitted.

Oh wow, some interesting stuff in here. Looks like they use MySQL as a queue for scheduling the ORC Peons? Would have loved to hear more about why they did that.

Author here. We use it because it works well for us. We've put a lot of work into making MySQL scale for us to the point where it's a very well supported system and one of the main choices for a lot of storage decisions. We even use MySQL as a queue for Facebook Messenger. More details about this: https://www.youtube.com/watch?v=eADBCKKf8PA https://code.facebook.com/posts/820258981365363/building-mob...

Thanks! Standardizing on MySQL because of internal expertise is definitely a great reason.

Re: 'Check Your Backups Work' Day

#87

Post by sytse »

mozair wrote: post #80
sytse wrote: post #67

Earlier quoted context omitted.

Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and…

make sure everyone is doing ok, don't let them beat themselves over it all the love and support in the world towards the team

Thanks mozair. Everyone is OK but sad. I sent this tweet earlier https://twitter.com/sytses/status/826598260831842308

Thanks for the support, we've received a lot of kind reactions and are very grateful for them.

Re: 'Check Your Backups Work' Day

#88

Post by andrewaylett »

PhantomGremlin wrote: post #44
aaronmdjones wrote: post #25

Earlier quoted context omitted.

Ouch. Doubly ouch that it seems that there's no confirmation dialog with a 5-second countdown before you can hit Yes, or whatever.

For critical stuff IMO there needs to be more than just a confirmation dialog. The user needs to be transitioned into a totally different state of mind from the usual click, click, click, click. E.g. forcing someone to manually type the characters D E L E T E before allowing deletion of something potentially important. Either that or everything should have multiple levels of undo. Everything.

There's certainly some stuff in AWS that requires you to type the name of the thing you're deleting into a text field in order to delete it, but I suspect that's purely a UI check -- there wouldn't be so much point in requiring the name twice in the underlying API.

So if you've got a dud client implementation then you're going to lose the check.

One way to do stuff like this is to have separate roles for read-only and read-write access. I pay a lot more attention to what I'm doing on the rare occasions I assume permissions to change things.

Re: 'Check Your Backups Work' Day

#90

Post by [deleted] »

OJFord wrote: post #79
hueving wrote: post #42

Earlier quoted context omitted.

Because it works perfectly fine? I tend to write anyone off who scoffs at simple database-as-queue designs without understanding what the scaling requirements are. You can use a database as a job queue for 10s of thousands of jobs per day without any sweat.

Scoffing? GP said it was "interesting stuff" and that they'd "love to hear more". What's wrong with that?

[deleted]
Post reply on HN