Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The absolute worst scenario happened

reddit.com

Post reply on HN
Print view View original

31–40 of 566 posts

Re: The absolute worst scenario happened

#31

Post by turbinerneiter »

brailsafe wrote: post #10
IgorPartola wrote: post #4

What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?

Was curious about that as well. I've never heard of such an obsurd amount of time, and it seems doubtful it would hold up. "You need to give us 3 months heads up no matter what, but don't worry, we don't have to give you any if you oversleep for a meeting."

It goes both ways, the employer has to give the employee 3 months notice as well.

But this is for when both of them can't find a mutual solution. You can always agree to terminate the contract earlier, if both parties agree.

At least that is how it works in Germany.

Re: The absolute worst scenario happened

#32

Post by teddyh »

TeeWEE wrote: post #22
teddyh wrote: post #20

They might check on robtex.com and other DNS historical information sites.

Exactly what i was thinking. Its D_NS its Distributed. The records are out there somewhere.

The ‘D’ in DNS stands for “Domain”, not “Distributed”.

Re: The absolute worst scenario happened

#33

Post by mnd999 »

IgorPartola wrote: post #4

What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?

They could not pay you.

Or

If your contract is exclusive they can pay you and prevent you from working for anywhere else (gardening leave).

You can’t (at least in most countries) enforce the contract in such a way that you force someone to work.

Re: The absolute worst scenario happened

#35

Post by scandox »

How people handle this level of disaster is always very revealing. In every case I've ever experienced someone kept their head and a solution was found. Usually there was reputational damage but surprisingly most times the actual business bounced back. But it was always the actions of someone in particular in those first hours that made the difference.

Thinking you're fucked is unhelpful, because you start talking about resumes and expecting some deus ex machina to appear and resolve the situation (for good or ill).

Re: The absolute worst scenario happened

#36

Post by Nextgrid »

ragnese wrote: post #23
Nextgrid wrote: post #3

This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains w…

Yeah, the poster didn't make totally clear whether all of their business records are totally gone, but it does sound like that's the case. The ERP was very likely the source of truth for everything- including employee information that the poster says is gone (can't send SMS to remote employees). So, it probably is that bad...

Nothing in his post suggests the ERP's DB is gone, merely that the mechanism which populates DNS records from the ERP has broken down, which probably has ripple effects on everything else as their internal DNS is broken and services can't talk to each other.

But the data should still be there, and it's a matter of reverse-engineering how to go from "ERP DB" to "DNS zone file".

Re: The absolute worst scenario happened

#37

Post by technics256 »

IgorPartola wrote: post #4

What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?

Depends on local law. In this case the user is in Finland under Finnish labor laws.

Re: The absolute worst scenario happened

#38

Post by ragnese »

Nextgrid wrote: post #36
ragnese wrote: post #23

Earlier quoted context omitted.

Yeah, the poster didn't make totally clear whether all of their business records are totally gone, but it does sound like that's the case. The ERP was very likely the source of truth for everything- including employee information that the poster says is gone (can't send SMS to remote employees). So, it probably is that bad...

Nothing in his post suggests the ERP's DB is gone, merely that the mechanism which populates DNS records from the ERP has broken down, which probably has ripple effects on everything else as their internal DNS is broken and services can't talk to each other. But the data should still be there, and it's a matter of reverse-engineering how to go from "ERP DB" to "DNS zone file".

Okay, fair enough. I probably skimmed the post too quickly.

Re: The absolute worst scenario happened

#39

Post by regularfry »

pedrovhb wrote: post #19
brailsafe wrote: post #5

Earlier quoted context omitted.

Didn't they only say that their backups are fucked, but not that they didn't try to make them? I've definitely run into situations where the business is only willing to pay for the most rudimentary conception of a backup strategy, and then it turned out to have been subject to corruption.

Can't find it now, but I've often seen a phrase to the effect of "If you don't regularly test that you can restore from backups, then you don't actually have backups", and it seems to ring quite true here.

It's up there with "unless you test power supply failover, don't rely on that generator to save you".

I know of one case where power failed, the "24 hour" generators kicked in, power company says "We'll need 12 hours" and the outage happened 6 hours later when they ran out of oil.

Re: The absolute worst scenario happened

#40

Post by PeterisP »

WJW wrote: post #2

That is an amazing story. Who knew that putting everything in a database without backups was risky?

They explicitly mention that they have backups, but that the "backups are fucked", whatever that means.

Ransomware scenarios come to mind, where the attackers will look for and actively destroy the backups you have if they are remotely accessible with e.g. AD admin credentials, and only trigger the ransomware when the backups are gone. If you have offsite backups in some cloud system, that helps you against natural disasters but not against malicious activity. The need for backups that are not just offsite but also offline is somewhat recent and very, very many companies do not have them.

Post reply on HN