Live data from Hacker News

The absolute worst scenario happened

reddit.com

71–80 of 566 posts

Re: The absolute worst scenario happened

#71

What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?

In America absolutely nothing happens. He might have some type of agreement where he gets a severance payment upon resignation, that would be forfeited. If I was him, assuming upper management starts yelling or whatever, I would just leave. Stress can kill you, and stressing over a job is absolutely never worth it because if you die today , they'll post a req for your position by the end of business.

It looks like they could probably sort something out, wouldn't be a bad idea to call up any of the old vets who retired to see if they could come down to the office and contract for a week or so.

Re: The absolute worst scenario happened

#72
post #15

improved link: https://old.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...

Thanks. Reddit is such a good example of how "progress" hasn't really been progress in web development. The new front-end framework site is so much slower and more buggy.

wow, it was so much better before. What the heck? Who made the decision to switch?

Re: The absolute worst scenario happened

#73

> we don't even have a listing of our ~300 employees SMS numbers

This surprised me. If ever there was a use case for an organizational structure, this is surely it.. just have every manager reach out to their direct reports? Following the tree communication should happen at something like nlogn speed

Re: The absolute worst scenario happened

#74
post #3

This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains w…

I agree. My first reaction was, "I think I could fix this". Worst case this may be be proceeded by a forensic recovery of the data. This doesn't seem impossible at all unless there is some other factor at play like ransomware/physical damage.

Re: The absolute worst scenario happened

#76
post #17
post #6

Earlier quoted context omitted.

I guess if you are in the business of selling services that include custom domains that isn't quite as crazy as it initially sounds... You are, of course, assuming they have a backup and the backup can actually be restored and that the restore contains the required information.

It does say in the original post that: > Our backups are fucked. So I assume that they have already tried and they didn't work.

Early in my career (>30 years) ago I nuked a companies salary database (a missing $ in a shell script moved everything to the file i)

No problem they said - we have backups. They had three tapes.

First tape failed.

Second tape failed.

Third tape worked.

In retrospect that was a useful, if rather stressful, lesson.

Re: The absolute worst scenario happened

#77

What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?

Not sure what country the person works in. In addition it could be a situation of a small town / tech backwater and word getting around, regardless of the legal question, so quitting out of contract could get you blacklisted.

Ultimately though a person could not show up for any reason at all (e.g. hit by the proverbial bus) and you need to plan for that. Sounds like a company that just flies by the seat of its pants so they probably have that 3-month clause for that reason.

Re: The absolute worst scenario happened

#78
post #2

That is an amazing story. Who knew that putting everything in a database without backups was risky?

Didn't they only say that their backups are fucked, but not that they didn't try to make them? I've definitely run into situations where the business is only willing to pay for the most rudimentary conception of a backup strategy, and then it turned out to have been subject to corruption.

They said they knew they haven't been able to restore backups for 2 years. Which is another way of saying that they don't have backups at all.

Re: The absolute worst scenario happened

#79
post #61

Earlier quoted context omitted.

From a reply the poster made in that thread: "So, we have backups on an encrypted drive, which we can't access (authentication is done via our custom ERP software, which is down as well - in hindsight, a very bad idea). And the backups don't work. We can rebuild the DNS servers from scratch, and the records are probably still safely stored somewhere inone of the 3 different databases our ERP software uses, but we don…

> Elsewhere, they say that two years ago, they found they were unable to restore from backups... So their business actually failed a couple years ago, and they have only noticed this month.

They noticed 2 years ago, and the ship sank below the water line this month.

Re: The absolute worst scenario happened

#80
post #35

How people handle this level of disaster is always very revealing. In every case I've ever experienced someone kept their head and a solution was found. Usually there was reputational damage but surprisingly most times the actual business bounced back. But it was always the actions of someone in particular in those first hours that made the difference. Thinking you're fucked is unhelpful, because you start talking ab…

One thing the situation does show is a lack of proper care for their most critical assets. When the dust settles, there should be hard questions for several people. One being-- when was the last time backups were tested..
Post reply on HN