Technical debt can lead to non-technical bankruptcy.
The absolute worst scenario happened
21–30 of 566 posts
Re: The absolute worst scenario happened
#22They might check on robtex.com and other DNS historical information sites.
Exactly what i was thinking. Its D_NS its Distributed. The records are out there somewhere.
Re: The absolute worst scenario happened
#23This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains w…
Yeah, the poster didn't make totally clear whether all of their business records are totally gone, but it does sound like that's the case. The ERP was very likely the source of truth for everything- including employee information that the poster says is gone (can't send SMS to remote employees).
So, it probably is that bad...
Re: The absolute worst scenario happened
#24What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?
Was curious about that as well. I've never heard of such an obsurd amount of time, and it seems doubtful it would hold up. "You need to give us 3 months heads up no matter what, but don't worry, we don't have to give you any if you oversleep for a meeting."
This isn’t particularly crazy in Europe. It indeed goes both ways.
Re: The absolute worst scenario happened
#25Earlier quoted context omitted.
Didn't they only say that their backups are fucked, but not that they didn't try to make them? I've definitely run into situations where the business is only willing to pay for the most rudimentary conception of a backup strategy, and then it turned out to have been subject to corruption.
Can't find it now, but I've often seen a phrase to the effect of "If you don't regularly test that you can restore from backups, then you don't actually have backups", and it seems to ring quite true here.
I'd go one step further - actually restore the databases and check that you can bring up the relevant applications that use those databases.
Re: The absolute worst scenario happened
#26The managed service provider company built a custom DNS inside an ERP without backups or bootstrap strategy in place.
House-of-cards core infrastructure is a sign of negligence and OP should have reconsidered working there a long time ago.
Re: The absolute worst scenario happened
#27"Every person who knew how the custom DNS system worked has left the company years ago."
Keeping the people who know how the system works is really the last line of defense.
From, the point where such last person left without any compensating mechanism in place, the clock started ticking
Re: The absolute worst scenario happened
#28“let the boat sync.” Sure, now you start thinking of backups.
I loled, although non-native speaker who does similar mistakes all the time :-)
Re: The absolute worst scenario happened
#29What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?
Was curious about that as well. I've never heard of such an obsurd amount of time, and it seems doubtful it would hold up. "You need to give us 3 months heads up no matter what, but don't worry, we don't have to give you any if you oversleep for a meeting."
I had a 6 month notice period in one role, 3 months isn't uncommon for senior roles, in the UK at least.
Re: The absolute worst scenario happened
#30This is the snow on the tip of the iceberg. I'm sure there's plenty more where this came from.
There's so many things wrong with the scenario described, that I can't even begin to talk about it.
My experience is that folks never want to talk about backup or DR, as they are expensive, and the idea is that they should never be used.
No one ever has a problem with insurance premiums, though...