Live data from Hacker News

The absolute worst scenario happened

reddit.com

91–100 of 566 posts

Re: The absolute worst scenario happened

#91
post #79
post #61

Earlier quoted context omitted.

> Elsewhere, they say that two years ago, they found they were unable to restore from backups... So their business actually failed a couple years ago, and they have only noticed this month.

They noticed 2 years ago, and the ship sank below the water line this month.

They were already dead 2 years ago, the body just didn't realize it and kept moving until last Friday...

Re: The absolute worst scenario happened

#92
post #3

This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains w…

From a reply the poster made in that thread: "So, we have backups on an encrypted drive, which we can't access (authentication is done via our custom ERP software, which is down as well - in hindsight, a very bad idea). And the backups don't work. We can rebuild the DNS servers from scratch, and the records are probably still safely stored somewhere inone of the 3 different databases our ERP software uses, but we don…

I'd say if it was such a house of cards, my best bet is that this security chain is full of holes. Maybe the ERP used a fixed password, which can be guessed/bruteforced.

Re: The absolute worst scenario happened

#93

Tangent Alert! I have a 3 months resignation period, so I can't even leave... Maybe it's my (US) American experience, but being unable to quit a job strikes me as a terrible policy. This guy(gal?) is really required to submit 3 months notice or face some sort of repercussions (financial, I assume)? Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)? Not that the A…

> Does this 3 month period apply in reverse

Yes (in France at least) it is on both sides, unless both sides agree to shorten it

> face some sort of repercussions (financial, I assume)?

I'm speaking for France. This is a tricky subject, you almost can't get any repercussion. But it can sucks, like the employer can refuse to fire you and suspend your contract, in the mean time, you are not paid but are not allowed to work for someone else.

Re: The absolute worst scenario happened

#94

Earlier quoted context omitted.

Nothing in his post suggests the ERP's DB is gone, merely that the mechanism which populates DNS records from the ERP has broken down, which probably has ripple effects on everything else as their internal DNS is broken and services can't talk to each other. But the data should still be there, and it's a matter of reverse-engineering how to go from "ERP DB" to "DNS zone file".

> Skimmed through the MySQL and Postgres databases, no signs of anything domain-related. I'm betting it's all stored in the custom integrated [and encrypted] KV database.. -- https://old.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...

If the system is old (going by other replies, introduced in a deal 20 years ago), maybe they'll be able to brute-force the encryption...

Re: The absolute worst scenario happened

#95

Tangent Alert! I have a 3 months resignation period, so I can't even leave... Maybe it's my (US) American experience, but being unable to quit a job strikes me as a terrible policy. This guy(gal?) is really required to submit 3 months notice or face some sort of repercussions (financial, I assume)? Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)? Not that the A…

> Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)?

Of course.

Different notice periods for employers and employees would be outright illegal in most of Europe.

Re: The absolute worst scenario happened

#96

Tangent Alert! I have a 3 months resignation period, so I can't even leave... Maybe it's my (US) American experience, but being unable to quit a job strikes me as a terrible policy. This guy(gal?) is really required to submit 3 months notice or face some sort of repercussions (financial, I assume)? Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)? Not that the A…

European here. Having 3 months written in your contract (and in many countries by law, like France) is pretty common.

You can still quit if both parties agree. I've done it before.

The clause is there to avoid this type of scenario where a key employee just says fuck it and leave.

It works both ways. So a company can say fuck it and fire you just because they want to.

It's a fantastic system. It can be exploited, but usually works really well for employees and it gives you a huge safety net you can (usually) rely on.

Re: The absolute worst scenario happened

#97

Tangent Alert! I have a 3 months resignation period, so I can't even leave... Maybe it's my (US) American experience, but being unable to quit a job strikes me as a terrible policy. This guy(gal?) is really required to submit 3 months notice or face some sort of repercussions (financial, I assume)? Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)? Not that the A…

In Sweden (don't know where this company is located) it's usual with collective bargain agreement and if you've been employed for along time (5+ years) it's not uncommon with 3 months notice. It is usually accompanied by 6 months if your employer wants to let you go (but it's not easy if you're the employer to let people go). Most of the time, these periods are negotiated when someone resigns though.

Re: The absolute worst scenario happened

#98
post #80
post #35

How people handle this level of disaster is always very revealing. In every case I've ever experienced someone kept their head and a solution was found. Usually there was reputational damage but surprisingly most times the actual business bounced back. But it was always the actions of someone in particular in those first hours that made the difference. Thinking you're fucked is unhelpful, because you start talking ab…

One thing the situation does show is a lack of proper care for their most critical assets. When the dust settles, there should be hard questions for several people. One being-- when was the last time backups were tested..

The answer to the question is there

>We can't access the backups. Even if we could, we noticed two years ago when trying a restore from the backups, that it doesn't work. Booting the restored server leads to a kernel panic we couldn't figure out. Management said we don't have enough money to fix any of this.

Re: The absolute worst scenario happened

#99

Earlier quoted context omitted.

Perfectly common in large parts of Europe. I also have a 3 months notice period, which is pretty much the standard here in Sweden for tech workers or other highly-qualified occupations. For very senior positions, 6 months isn't unheard of. It goes both ways of course, my employer is also bound by the 3 month period in case they want to fire me.

How does getting a new job work? You tell the new company that you have to give 3 months notice? That's some long term planning on the part of the hiring company.

Don't forget that the notice period applies to all companies and all employees. So the whole culture expects that, and works with the long lead times.

Re: The absolute worst scenario happened

#100
post #86

Earlier quoted context omitted.

These types of situations always wind up being the most fun I have at work and leave me feeling most fulfilled. I wish I could do that all day instead of going to standups.

A "firefighting" consultancy might be fun, no? And you would give people a lot of time off!

I've thought about that before. You get the call. "It's 30 minutes away. I'll be there in 10"

If anyone happens to know of a career path or company that does that sort of work I would be interested to hear about it. Bonus points if the pay is half decent.

Post reply on HN