Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The absolute worst scenario happened

reddit.com

Post reply on HN
Print view View original

61–70 of 566 posts

Re: The absolute worst scenario happened

#61

Post by ansible »

rebuilder wrote: post #51
Nextgrid wrote: post #3

This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains w…

From a reply the poster made in that thread: "So, we have backups on an encrypted drive, which we can't access (authentication is done via our custom ERP software, which is down as well - in hindsight, a very bad idea). And the backups don't work. We can rebuild the DNS servers from scratch, and the records are probably still safely stored somewhere inone of the 3 different databases our ERP software uses, but we don…

> Elsewhere, they say that two years ago, they found they were unable to restore from backups...

So their business actually failed a couple years ago, and they have only noticed this month.

Re: The absolute worst scenario happened

#62

Post by colmvp »

scandox wrote: post #35

How people handle this level of disaster is always very revealing. In every case I've ever experienced someone kept their head and a solution was found. Usually there was reputational damage but surprisingly most times the actual business bounced back. But it was always the actions of someone in particular in those first hours that made the difference. Thinking you're fucked is unhelpful, because you start talking ab…

Thinking you're fucked is unhelpful, but it's pretty normal to go full Alien Hudson in a dire situation and need someone else to assure you that things will be okay.

Re: The absolute worst scenario happened

#64

Post by elif »

ajb wrote: post #16
IgorPartola wrote: post #4

What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?

I guess it could be a problem in the US for COBRA (health insurance continuation)? apart from that, if the company is dead I guess there's little they can take away that you aren't going to lose anyway. [edited to add : not relevant, they're in Finland: https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu... ]

COBRA still applies if you leave voluntarily.

Source: left voluntarily

Re: The absolute worst scenario happened

#65

Post by fuzzy2 »

PeterisP wrote: post #40
WJW wrote: post #2

That is an amazing story. Who knew that putting everything in a database without backups was risky?

They explicitly mention that they have backups, but that the "backups are fucked", whatever that means. Ransomware scenarios come to mind, where the attackers will look for and actively destroy the backups you have if they are remotely accessible with e.g. AD admin credentials, and only trigger the ransomware when the backups are gone. If you have offsite backups in some cloud system, that helps you against natural d…

They backups are probably fine (as in “the data is in there”) but there may be no plans on how to restore them. Or they may be incomplete.

I doubt any external factors are to blame.

Re: The absolute worst scenario happened

#66

Post by Semaphor »

Nextgrid wrote: post #36
ragnese wrote: post #23

Earlier quoted context omitted.

Yeah, the poster didn't make totally clear whether all of their business records are totally gone, but it does sound like that's the case. The ERP was very likely the source of truth for everything- including employee information that the poster says is gone (can't send SMS to remote employees). So, it probably is that bad...

Nothing in his post suggests the ERP's DB is gone, merely that the mechanism which populates DNS records from the ERP has broken down, which probably has ripple effects on everything else as their internal DNS is broken and services can't talk to each other. But the data should still be there, and it's a matter of reverse-engineering how to go from "ERP DB" to "DNS zone file".

> Skimmed through the MySQL and Postgres databases, no signs of anything domain-related. I'm betting it's all stored in the custom integrated [and encrypted] KV database..

-- https://old.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...

Re: The absolute worst scenario happened

#67

Post by 4e530344963049 »

scandox wrote: post #35

How people handle this level of disaster is always very revealing. In every case I've ever experienced someone kept their head and a solution was found. Usually there was reputational damage but surprisingly most times the actual business bounced back. But it was always the actions of someone in particular in those first hours that made the difference. Thinking you're fucked is unhelpful, because you start talking ab…

If you have been that person before, it can be fun and exciting to think up solutions and implement them under pressure. Definitely a real test in your ability to concentrate. But don't expect the credit you deserve for it, and don't even expect others to even bother to put in any effort towards fixing things.

Also, make sure everyone knows you are going on vacation once you resolve things.

Re: The absolute worst scenario happened

#68

Post by arethuza »

PeterisP wrote: post #50
arethuza wrote: post #29

Earlier quoted context omitted.

I had a 6 month notice period in one role, 3 months isn't uncommon for senior roles, in the UK at least.

What would be the practical consequences to the employee if they break that notice period in UK, i.e. stop coming to work (or, especially now with Covid, simply stop doing the remote work) sooner than these 3 months have passed?

I suspect you could be sued by your employer. Mind you I've never heard of anyone being sued in that way - can't imagine it would be worth it for the employer unless they could demonstrate a significant financial loss.

Re: The absolute worst scenario happened

#69

Post by ACS_Solver »

brailsafe wrote: post #10
IgorPartola wrote: post #4

What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?

Was curious about that as well. I've never heard of such an obsurd amount of time, and it seems doubtful it would hold up. "You need to give us 3 months heads up no matter what, but don't worry, we don't have to give you any if you oversleep for a meeting."

Perfectly common in large parts of Europe. I also have a 3 months notice period, which is pretty much the standard here in Sweden for tech workers or other highly-qualified occupations. For very senior positions, 6 months isn't unheard of.

It goes both ways of course, my employer is also bound by the 3 month period in case they want to fire me.

Re: The absolute worst scenario happened

#70

Post by viraptor »

arnaudsm wrote: post #63

Are the CTO and sysadmins liable in such scenarios? Can they be sued ?

Anyone can be sued. But it wouldn't be a good idea if they kept the receipts:

https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...

> We tried really hard to make management aware and actually succeeded in that, only for management to follow up and say we don't have enough money to fix it, and they understood the possible outcomes.

Post reply on HN