51–60 of 152 posts
No hidding, no eupheminization, their live doc stream actually made me question what things did I do on my systems. Looks like convergent evolution in some parts, like prompt changes.
Thanks Gitlab.
Earlier quoted context omitted.
You know, I remember reading years ago that during the launch preparation for Apollo 16(?), they did a routine pressure test of the Command Module atop the rocket a few days before launch. The technician in charge made the simple error of forgetting to open a pressure release valve during the test which lead to an overpressurisation of the CM causing significant damage, including separation of the heat shield. The EN…
>When the Apollo launchpad manager was asked if he had fired the technician in question, then answer was allegedly "Nope. He is the one guy on the next launch team that I know will NEVER make the same mistake again." This same general story appears in different forms with million dollar trading errors, etc. But I have to wonder if it's really that good of a lesson. If you truly take it to heart, you could potentially…
Make a backup, restore to test environment, run checksums, anonymize, release test environment.
That way each and every backup is tested both for integrity and ability to rebuild a working environment from it.
In my practice insufficient backup is still (unfortunately) a very common occurrence.
On another note: just having stuff stored with triple replication in the cloud is emphatically NOT a backup.
And it also helps if the same people that have access to the live environment do not have write access to the backups, but that's only feasible past a certain team size.
Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…
> Imagine if you're the CEO of Gitlab and seeing this right now I kind of hope the CEO of Gitlab isn't reading HN right now
Earlier quoted context omitted.
You know, I remember reading years ago that during the launch preparation for Apollo 16(?), they did a routine pressure test of the Command Module atop the rocket a few days before launch. The technician in charge made the simple error of forgetting to open a pressure release valve during the test which lead to an overpressurisation of the CM causing significant damage, including separation of the heat shield. The EN…
>When the Apollo launchpad manager was asked if he had fired the technician in question, then answer was allegedly "Nope. He is the one guy on the next launch team that I know will NEVER make the same mistake again." This same general story appears in different forms with million dollar trading errors, etc. But I have to wonder if it's really that good of a lesson. If you truly take it to heart, you could potentially…
Everyone is talking about backups, but why not about this? How is it even possible to delete the production database by accident? Why does he have SSH access there? Why do they test their database replication in production? Why are they fire-fighting by changing random database settings _in production_?
I know that all of this is common practice. I am questioning it in general.
I think having a day for a whole year is a bit sparse. Some startups start and shutdown within a year. Apart from checking backups after any big code change related to backups, I think backups should be checked quarterly. It takes no more than couple hours most of the time, and as our wise said, "An ounce of prevention is worth a pound of cure"
I think that once a quarter is better, but if your start up shuts down at the end of the year, you probably don't need to worry about it
When stuff "just works", you don't need to check your backups. I fully trust my iPhone's iCloud backups, my Time Machine backups, and my cloud rsyncs. Time Machine also lets me know if they get corrupted, or if I haven't backed up in a while. That's how backups should work - an adage of "you don't have backups unless you check them" just won't work for most people.