Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

'Check Your Backups Work' Day

checkyourbackups.work

Post reply on HN
Print view View original

141–150 of 152 posts

Re: 'Check Your Backups Work' Day

#141

Post by AsyncAwait »

soheil wrote: post #21

Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…

Indeed, I just started planning for a side project to solve this by automating as much of the backup sanity checking procedure as possible.

Re: 'Check Your Backups Work' Day

#142

Post by AsyncAwait »

throwaway1236t4 wrote: post #115
sytse wrote: post #67

Earlier quoted context omitted.

Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and…

I also would ask Why are we running our own Postgre setup and not using RDS? And then Why do we not have production Postgre DBAs on staff that would do this rather than an engineer?

They want to have more control over the database and moving providers etc., which is why they're not using RDS, as was explained on the live stream.

Re: 'Check Your Backups Work' Day

#143

Post by AsyncAwait »

erikbye wrote: post #132
ocdtrekkie wrote: post #120

Earlier quoted context omitted.

Hindsight is always 20/20. Berating people who now KNOW mistakes were made accomplishes nothing.

Backup is elementary. It should not need to be learned nor manifest itself as a hindsight revelation. Frankly GitLab seems more and more like a mom and pop shop that learned Rails over a weekend. They did not even know if they had functioning backups before shit hit the fan.

I bet you're making as successful of a software as they are. Don't answer...

Re: 'Check Your Backups Work' Day

#144

Post by paulddraper »

ktta wrote: post #133
paulddraper wrote: post #9

Earlier quoted context omitted.

I think that once a quarter is better, but if your start up shuts down at the end of the year, you probably don't need to worry about it

I don't think any startup knows that they're going to shutdown within the year. No one would take the time if they knew they were going to shut down soon.

Correct. So save some concerns (e.g. weekly verification of backups) until after a year.

Re: 'Check Your Backups Work' Day

#145

Post by hueving »

safeaim wrote: post #54
hueving wrote: post #41

Earlier quoted context omitted.

>When the Apollo launchpad manager was asked if he had fired the technician in question, then answer was allegedly "Nope. He is the one guy on the next launch team that I know will NEVER make the same mistake again." This same general story appears in different forms with million dollar trading errors, etc. But I have to wonder if it's really that good of a lesson. If you truly take it to heart, you could potentially…

And if one starts applying your principle at a workplace, you end up with the whole company being afraid of doing anything at all, to avoid being fired.

No, there is a level of competency where once you make mistakes below that, firing is the right move.

If a surgeon keeps killing patients because he keeps forgetting to wash his hands, then he needs to be fired.

If a developer skips all company policies and deploys directly to production without a good reason, then he needs to be fired.

There are reasonable mistakes, then there are errors resulting from wreckless actions indicative of a larger problem with the person's view towards their work.

Re: 'Check Your Backups Work' Day

#146

Post by cyberferret »

bluGill wrote: post #126
hueving wrote: post #41

Earlier quoted context omitted.

>When the Apollo launchpad manager was asked if he had fired the technician in question, then answer was allegedly "Nope. He is the one guy on the next launch team that I know will NEVER make the same mistake again." This same general story appears in different forms with million dollar trading errors, etc. But I have to wonder if it's really that good of a lesson. If you truly take it to heart, you could potentially…

That is the wrong question. The real question is what did they do to ensure that this mistake cannot be made again by anybody. In the form this variation is in, the real question is what changes were made to the design so that the system cannot be over pressurized in that way by mistake.

Sometimes you can put in all the checks and balances you wish, but humans will just make a, well, human mistake.

All commercial aircraft have checklists in place, and a two person cross checking system in the cockpit, as well as mechanical means to detect potential problems - but aircraft still every now and then land with their gear still up etc.

Does it make economic sense for an organisation that has invested possibly millions of dollars in training a pilot to immediately sack him/her for a costly mistake like this? Sure, disciplinary action and possible demotion within the ranks is a given, but will you get rid of the one person who will be almost fanatical about never landing with the gear up again?

If they consistently make technical mistake and are sloppy in their airmanship, then sure, fire away. But it is going to be pretty much impossible to design a 'system' that can prevent human errors from creeping in.

Re: 'Check Your Backups Work' Day

#148

Post by chiefalchemist »

korzun wrote: post #131
michaelt wrote: post #102

Earlier quoted context omitted.

I've never heard of anyone successfully suing AWS/Google/Azure over lost data or downtime. And presumably even if your notes are kept private, it'll still come out in discovery if you end up in court? What are the risks here? I'd be interested to learn more.

> I've never heard of anyone successfully suing AWS/Google/Azure over lost data or downtime. Doesn't mean it never happens.

Imagine how deep their pockets are. Now imagine how deep your pockets would need to be to go toe to toe with them.

Sure plenty of uber larges Incs are "in the cloud" but with mission critical (read: high value worth sueing over) applications and data? That's a pretty small universe.

Re: 'Check Your Backups Work' Day

#149

Post by dspillett »

marcosdumay wrote: post #139
dspillett wrote: post #135

Earlier quoted context omitted.

> Really, how can you be sure your restoration tests are running correctly and their messages get all the way to you? I get a daily message, so if I know if things aren't getting sent or getting through. I've been wary of setting up anything that only sends problem messages because regularly getting the "all OK" messages proves that at least that part of the infrastructure is working.

The problem with that is when you start getting too many "all OK" messages. Getting a message must be mostly a surprise. One daily message is OK for me, but if so it may be good to tell about more than a backup. A daily digest of "everything is up and running" may indeed be something very nice to have. And how do you ensure a team will always have someone surprised not to get a daily message? Maybe sending for may pe…

That is why I'm thinking of scripting up something to scan the status messages and present a list (in the console or on a web page), colour highlighted (normal for info only, green for all OK, red for error or missed message, big red bold (flashing?) for an issue still not resolved from the previous day. That automates checking but I've still got the individual updates in case that fails. Maybe I'll make the output of this my homepage and/or console login banner.

> And how do you ensure a team will always have someone surprised not to get a daily message?

For a team you have a management problem not a technical one! Somebody needs to be responsible for it in the end, plain and simple.

On a less authoritarian note a technical helper might perhaps be to have a screen, always on, showing a dashboard of statuses, where everyone can see it? Make it beep incessantly when a new warning arrives until someone acknowledges it. If you acknowledge a message either deal with it (if you can) or notify whoever needs to be told so they can take action.

People will still sometimes ignore or miss the warnings. Nothing and no one is perfect.

But I'm only dealing with my own collection of personal data and projects here so it isn't that scale of problem - at work I used to be responsible for infrastructure as well as my actual job but we no longer operate at a scale where that is at all practical so I have to trust someone else is dealing with it (though I'm not that trusting I do sometimes take time out to check things myself where I have access).

Re: 'Check Your Backups Work' Day

#150

Post by sumbry »

sytse wrote: post #67
edelans wrote: post #62

Earlier quoted context omitted.

Agreed ! Even when they screw things up, they make people benefit from it! Kudos for such a good spirit and dedication to your customers. Everybody screw up from time to time in our industry. And when this happens, you have 2 types of guy : those who try to hide it, and those like Giltlab team who communicate as fast as they can because they respect their customers. Paradoxically, to me, it creates more trust than it…

Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and…

If you do the 5 why's and are honest you'll discover the real root cause is not having a backup and restore procedure that is programmatic.

All outages are blameless. It's always a process failure or lack of a proper system or tool.

Post reply on HN