61–70 of 621 posts
I'm not complaining, but backup/restore is important part, with 100% test coverage and daily backup/restore runs.
followed by
> So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place.
is no way to be running a public service with customer data. Did the person who set up that S3 job simply write a script or something and just go "yep, it's done" and walk away? Seriously?
Earlier quoted context omitted.
Does this mean whatever was in that database is gone, with no available backups? Is this an SOA where important data might lie in another service or data store, or is this a monolithic app and DB that is responsible for many (or all) things? What was stored in that database? Does this affect user data? Code?
We have snapshots, but they're not very recent (see the document for more info). The most recent snapshot is roughly 6 hours old (relative to the data loss). The data loss only affects database data, Git repositories and Wikis still exist (though they are fairly useless without a corresponding project).
This is the stuff my nightmares consist of after 900 consecutive days of being on call (and counting).
Are you a one man team or...? My wife would probably leave me if I was on-call for that long.
But the constant nagging in the back of your head that shit can go wrong at any second is draining and has been the biggest stressor in my life for a long time now.
My S.O. still gets mildly upset when I pack up the laptop on our way out to a fancy dinner, or disappear with my laptop when visiting her parents, but the fact that our life goals are aligned is the saving grace of all these situations. We both know what we want out of the next 5 years of our lives and are willing to sacrifice to achieve this goal (long term financial security).
I noticed the issue when I was pushing code earlier today. Hopefully this gets resolved soon. You guys are doing a great job. Keep up the good work!
And we're sorry for the inconvenience this caused to your workflow today!
Ouch. That is so harsh. Sorry to hear about the incident. Testing ones backups can be a pain to do but it is so very important.
> Our backups to S3 apparently don’t work either: the bucket is empty followed by > So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. is no way to be running a public service with customer data. Did the person who set up that S3 job simply write a script or something and just go "yep, it's done" and walk away? Seriously?
> Did the person who set up that S3 job simply write a
> script or something and just go "yep, it's done" and
> walk away?
23:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.
Good lesson on making command prompts on machines always tell you exactly what machine you're working on.
uname -n
Takes seconds.