Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

'Check Your Backups Work' Day

checkyourbackups.work

Post reply on HN
Print view View original

21–30 of 152 posts

Re: 'Check Your Backups Work' Day

#21

Post by soheil »

Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time and even then it could still just stop working one day.

We should probably treat this issue as something more like a disease like high blood pressure, namely, you don't know you have it, but it is probably doing irreparable damage to your internal organs. If we had no name for the disease or understanding of it we would just die at an earlier age without an obvious cause.

Let's identify the diseases of this sort in our industry and work on prevention, diagnosis and treatment of them instead of just saying you should have been more careful.

Re: 'Check Your Backups Work' Day

#22

Post by ParadisoShlee »

soheil wrote: post #21

Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…

I have nothing, but respect for the GitLab team.... offering live notes for the recovery is stunning.

Re: 'Check Your Backups Work' Day

#23

Post by yeukhon »

One time at work I accidentally triggered delete on an RDS CloudFormation stack It was not fun. Automatic backup from AWS is useless because automatic snapshots are removed as soon as the RDS instance is removed unless you tell AWS to make a final snapshot. We didn't have that flag in the stack template at the time, so ugh.

Oh how I deleted the stack? I was using the mobile app and was trying to look at the status of the Cfn stack, but the app was laggy my finger pressed the wrong button... sigh The other interesting thing was I checked the status because the previous night I changed my RDS instance to provisioned IOPS (took 8 hours and failed too). I felt sad and guilty, but at the same time I felt whatever because the upgrade didn't go through so perhaps this accident was all meant to be....

Re: 'Check Your Backups Work' Day

#24

Post by Andys »

riffraff wrote: post #20
scurvy wrote: post #16

Earlier quoted context omitted.

It's in there now. Row counts per table and checksums are written out with the backup for every table. If things don't match up, alarm in a loud and noisy way.

The point is that you could have a bug in the verification process that returns "all good" when it really isn't.

I have a SELECT of the most recent updated_at fields for a few tables emailed to me so I can eyeball the values

Re: 'Check Your Backups Work' Day

#25

Post by aaronmdjones »

yeukhon wrote: post #23

One time at work I accidentally triggered delete on an RDS CloudFormation stack It was not fun. Automatic backup from AWS is useless because automatic snapshots are removed as soon as the RDS instance is removed unless you tell AWS to make a final snapshot. We didn't have that flag in the stack template at the time, so ugh. Oh how I deleted the stack? I was using the mobile app and was trying to look at the status of…

Ouch.

Doubly ouch that it seems that there's no confirmation dialog with a 5-second countdown before you can hit Yes, or whatever.

Re: 'Check Your Backups Work' Day

#27

Post by yeukhon »

aaronmdjones wrote: post #25
yeukhon wrote: post #23

One time at work I accidentally triggered delete on an RDS CloudFormation stack It was not fun. Automatic backup from AWS is useless because automatic snapshots are removed as soon as the RDS instance is removed unless you tell AWS to make a final snapshot. We didn't have that flag in the stack template at the time, so ugh. Oh how I deleted the stack? I was using the mobile app and was trying to look at the status of…

Ouch. Doubly ouch that it seems that there's no confirmation dialog with a 5-second countdown before you can hit Yes, or whatever.

Ever since I removed the app from my phone. It's buggy and right, I really do expect them to have a dialog both on PC and mobile...

Re: 'Check Your Backups Work' Day

#28

Post by yeukhon »

soheil wrote: post #21

Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…

I think this is a big slap to company when they need to talk to their enterprise customers. I am not sure how we can really prevent this from happening by implementing an industry standard.

Re: 'Check Your Backups Work' Day

#29

Post by shubhamjain »

The graver problem is programmers — and, humans in general — don't realise the gravity of recklessness until the moment when shit hits the fan. Startups begin with general imprudence towards checks and processes, which is understandable but even in the growing stages, the idea of incorporating them is snubbed because of 'priorities'. "Time is precious and there are more important problems to solve in the way".

Unauthenticated MongoDb on default port? The likelihood of a person port scanning the entire web just doesn't strike as anything adverse, and then, someday someone does exactly that. I guess this is one important reason to bring at least some experienced people on board because there is a likely chance that they can give a better perspective on seriousness of such issues.

Post reply on HN