Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The absolute worst scenario happened

reddit.com

Post reply on HN
Print view View original

11–20 of 566 posts

Re: The absolute worst scenario happened

#11

Post by IgorPartola »

Nextgrid wrote: post #3

This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains w…

We don’t know that the records are still in the database. This could be a matter of ON DELETE CASCADE followed by a delete of what looked like innocent data and then boom it’s all gone. Then the integration updates bind which helpfully erases all records and then once there are no domain names to talk to everything is down.

I would be looking at the Internet Archive and Whois too figure out some basic domains and what the registrars are and go from there but of course depends on what business they are in and what their domain names were doing.

Re: The absolute worst scenario happened

#12

Post by matsemann »

IgorPartola wrote: post #4

What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?

If the company is about to go insolvent it would probably be happy about not having to pay for three months. Those three months usually go both ways.

But I think it would be better to keep it above board. Present it as a solution, kinda. Like, "we both want the same thing here".

Re: The absolute worst scenario happened

#14

Post by le-mark »

It’s easy to imagine there are a lot of these type of time bombs out there, particularly in really old legacy systems (> 20 years for example). I was at one place where even building the application for deployment led to a one week outage when some core people were laid off.

Re: The absolute worst scenario happened

#16

Post by ajb »

IgorPartola wrote: post #4

What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?

I guess it could be a problem in the US for COBRA (health insurance continuation)? apart from that, if the company is dead I guess there's little they can take away that you aren't going to lose anyway.

[edited to add : not relevant, they're in Finland: https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu... ]

Re: The absolute worst scenario happened

#17

Post by WJW »

arethuza wrote: post #6
Nextgrid wrote: post #3

This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains w…

I guess if you are in the business of selling services that include custom domains that isn't quite as crazy as it initially sounds... You are, of course, assuming they have a backup and the backup can actually be restored and that the restore contains the required information.

It does say in the original post that:

> Our backups are fucked.

So I assume that they have already tried and they didn't work.

Re: The absolute worst scenario happened

#19

Post by pedrovhb »

brailsafe wrote: post #5
WJW wrote: post #2

That is an amazing story. Who knew that putting everything in a database without backups was risky?

Didn't they only say that their backups are fucked, but not that they didn't try to make them? I've definitely run into situations where the business is only willing to pay for the most rudimentary conception of a backup strategy, and then it turned out to have been subject to corruption.

Can't find it now, but I've often seen a phrase to the effect of "If you don't regularly test that you can restore from backups, then you don't actually have backups", and it seems to ring quite true here.
Post reply on HN