The absolute worst scenario happened
reddit.com
The absolute worst scenario happened
1–10 of 566 posts
Re: The absolute worst scenario happened
#2That is an amazing story. Who knew that putting everything in a database without backups was risky?
Re: The absolute worst scenario happened
#3This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains which should at least bring their internal services back online and allow them to proceed further.
Re: The absolute worst scenario happened
#4What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?
Re: The absolute worst scenario happened
#5That is an amazing story. Who knew that putting everything in a database without backups was risky?
Didn't they only say that their backups are fucked, but not that they didn't try to make them? I've definitely run into situations where the business is only willing to pay for the most rudimentary conception of a backup strategy, and then it turned out to have been subject to corruption.
Re: The absolute worst scenario happened
#6This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains w…
I guess if you are in the business of selling services that include custom domains that isn't quite as crazy as it initially sounds...
You are, of course, assuming they have a backup and the backup can actually be restored and that the restore contains the required information.
Re: The absolute worst scenario happened
#7I’ve seen a few data disasters in my time but I think this takes the cake. It’s a good reminder of why you just need to prioritise the sort of work that has no immediate payoff.
Re: The absolute worst scenario happened
#8How many people had to log into their own work systems and check that things were running in order to be sure it wasn’t their employer?
Re: The absolute worst scenario happened
#9This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains w…
> > All domain information was wiped out and records became null.
> > Every person who knew how the custom DNS system worked has left the company years ago. Our backups are fucked. Our records are wiped from all domain servers out there.
Sure sounds like the data is lost to me.
Re: The absolute worst scenario happened
#10What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?
Was curious about that as well. I've never heard of such an obsurd amount of time, and it seems doubtful it would hold up.
"You need to give us 3 months heads up no matter what, but don't worry, we don't have to give you any if you oversleep for a meeting."