Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The absolute worst scenario happened

reddit.com

Post reply on HN
Print view View original

41–50 of 566 posts

Re: The absolute worst scenario happened

#41

Post by arethuza »

regularfry wrote: post #39
pedrovhb wrote: post #19

Earlier quoted context omitted.

Can't find it now, but I've often seen a phrase to the effect of "If you don't regularly test that you can restore from backups, then you don't actually have backups", and it seems to ring quite true here.

It's up there with "unless you test power supply failover, don't rely on that generator to save you". I know of one case where power failed, the "24 hour" generators kicked in, power company says "We'll need 12 hours" and the outage happened 6 hours later when they ran out of oil.

I contracted at one place for a bit where they shut everything down in each of their data centres once a year and power everything back up.

When I was there this didn't go too well and they couldn't get one their data centres online again - failover to their other centres did work though. This was ~20 years ago in the finance sector.

Re: The absolute worst scenario happened

#43

Post by ewindal »

brailsafe wrote: post #5
WJW wrote: post #2

That is an amazing story. Who knew that putting everything in a database without backups was risky?

Didn't they only say that their backups are fucked, but not that they didn't try to make them? I've definitely run into situations where the business is only willing to pay for the most rudimentary conception of a backup strategy, and then it turned out to have been subject to corruption.

Backups were extant, but inaccessible, and non-functional due to causing a kernel panic when applied. [1]

[1] https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...

Re: The absolute worst scenario happened

#45

Post by protomyth »

Well, it looks like someone better track down the people who knew how it worked and offer some serious cash.

The only actual backup is one that has been tested. With the cost of hardware, buy 2 or have an agreement with your vendor to have a machine ready to receive a backup.

Re: The absolute worst scenario happened

#46

Post by erikstarck »

This reminds me of when the Danish mega-corp Maersk was attacked by ransomware and had all of their computers locked and decrypted with no access.

All computers except one, which due to a power failure had been offline the whole time.

This computer had the critical DNS information needed to restore the network.

Only problem: it was in Africa, in a country that required Visa which would take weeks to apply for and receive. So someone from the African office had to bring the hard drive to an airport that someone from the London office could travel to and pick up the critical hard drive, then bring it on a plane back to London.

A sweaty and nervous trip, I can imagine.

Whole story here: https://www.wired.com/story/notpetya-cyberattack-ukraine-rus...

Re: The absolute worst scenario happened

#47

Post by xchaotic »

arethuza wrote: post #29
brailsafe wrote: post #10

Earlier quoted context omitted.

Was curious about that as well. I've never heard of such an obsurd amount of time, and it seems doubtful it would hold up. "You need to give us 3 months heads up no matter what, but don't worry, we don't have to give you any if you oversleep for a meeting."

I had a 6 month notice period in one role, 3 months isn't uncommon for senior roles, in the UK at least.

while I agree that this is common in Europe, if the company ceases to exist, what good is the 3 months notice for the employee? If he/she has any options, I'd leave asap and not worry about amicable terms with a soon to be non-existent entity.

Re: The absolute worst scenario happened

#48

Post by [deleted] »

ragnese wrote: post #23
Nextgrid wrote: post #3

This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains w…

Yeah, the poster didn't make totally clear whether all of their business records are totally gone, but it does sound like that's the case. The ERP was very likely the source of truth for everything- including employee information that the poster says is gone (can't send SMS to remote employees). So, it probably is that bad...

[deleted]

Re: The absolute worst scenario happened

#49

Post by TrackerFF »

IgorPartola wrote: post #4

What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?

Well, for one, your work references probably won't be the best.

Even if you're not responsible for the problem itself, leaving your employer in the ditch, when there's a major outage - well, it just doesn't look good (barring some scenario where your employer is completely unreasonable - like not paying you).

It's kinda like a firefighter that decides to turn off the hose and go home, in the middle of a raging fire. Who in their right mind would want to hire him/her in the future?

Re: The absolute worst scenario happened

#50

Post by PeterisP »

arethuza wrote: post #29
brailsafe wrote: post #10

Earlier quoted context omitted.

Was curious about that as well. I've never heard of such an obsurd amount of time, and it seems doubtful it would hold up. "You need to give us 3 months heads up no matter what, but don't worry, we don't have to give you any if you oversleep for a meeting."

I had a 6 month notice period in one role, 3 months isn't uncommon for senior roles, in the UK at least.

What would be the practical consequences to the employee if they break that notice period in UK, i.e. stop coming to work (or, especially now with Covid, simply stop doing the remote work) sooner than these 3 months have passed?
Post reply on HN