Earlier quoted context omitted.
> Elsewhere, they say that two years ago, they found they were unable to restore from backups... So their business actually failed a couple years ago, and they have only noticed this month.
They noticed 2 years ago, and the ship sank below the water line this month.
The absolute worst scenario happened
91–100 of 566 posts
Re: The absolute worst scenario happened
#92This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains w…
From a reply the poster made in that thread: "So, we have backups on an encrypted drive, which we can't access (authentication is done via our custom ERP software, which is down as well - in hindsight, a very bad idea). And the backups don't work. We can rebuild the DNS servers from scratch, and the records are probably still safely stored somewhere inone of the 3 different databases our ERP software uses, but we don…
Re: The absolute worst scenario happened
#93Tangent Alert! I have a 3 months resignation period, so I can't even leave... Maybe it's my (US) American experience, but being unable to quit a job strikes me as a terrible policy. This guy(gal?) is really required to submit 3 months notice or face some sort of repercussions (financial, I assume)? Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)? Not that the A…
Yes (in France at least) it is on both sides, unless both sides agree to shorten it
> face some sort of repercussions (financial, I assume)?
I'm speaking for France. This is a tricky subject, you almost can't get any repercussion. But it can sucks, like the employer can refuse to fire you and suspend your contract, in the mean time, you are not paid but are not allowed to work for someone else.
Re: The absolute worst scenario happened
#94Earlier quoted context omitted.
Nothing in his post suggests the ERP's DB is gone, merely that the mechanism which populates DNS records from the ERP has broken down, which probably has ripple effects on everything else as their internal DNS is broken and services can't talk to each other. But the data should still be there, and it's a matter of reverse-engineering how to go from "ERP DB" to "DNS zone file".
> Skimmed through the MySQL and Postgres databases, no signs of anything domain-related. I'm betting it's all stored in the custom integrated [and encrypted] KV database.. -- https://old.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
Re: The absolute worst scenario happened
#95Tangent Alert! I have a 3 months resignation period, so I can't even leave... Maybe it's my (US) American experience, but being unable to quit a job strikes me as a terrible policy. This guy(gal?) is really required to submit 3 months notice or face some sort of repercussions (financial, I assume)? Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)? Not that the A…
Of course.
Different notice periods for employers and employees would be outright illegal in most of Europe.
Re: The absolute worst scenario happened
#96Tangent Alert! I have a 3 months resignation period, so I can't even leave... Maybe it's my (US) American experience, but being unable to quit a job strikes me as a terrible policy. This guy(gal?) is really required to submit 3 months notice or face some sort of repercussions (financial, I assume)? Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)? Not that the A…
You can still quit if both parties agree. I've done it before.
The clause is there to avoid this type of scenario where a key employee just says fuck it and leave.
It works both ways. So a company can say fuck it and fire you just because they want to.
It's a fantastic system. It can be exploited, but usually works really well for employees and it gives you a huge safety net you can (usually) rely on.
Re: The absolute worst scenario happened
#97Tangent Alert! I have a 3 months resignation period, so I can't even leave... Maybe it's my (US) American experience, but being unable to quit a job strikes me as a terrible policy. This guy(gal?) is really required to submit 3 months notice or face some sort of repercussions (financial, I assume)? Does this 3 month period apply in reverse (employer can't fire employees without a full 3 month warning)? Not that the A…
Re: The absolute worst scenario happened
#98How people handle this level of disaster is always very revealing. In every case I've ever experienced someone kept their head and a solution was found. Usually there was reputational damage but surprisingly most times the actual business bounced back. But it was always the actions of someone in particular in those first hours that made the difference. Thinking you're fucked is unhelpful, because you start talking ab…
One thing the situation does show is a lack of proper care for their most critical assets. When the dust settles, there should be hard questions for several people. One being-- when was the last time backups were tested..
>We can't access the backups. Even if we could, we noticed two years ago when trying a restore from the backups, that it doesn't work. Booting the restored server leads to a kernel panic we couldn't figure out. Management said we don't have enough money to fix any of this.
Re: The absolute worst scenario happened
#99Earlier quoted context omitted.
Perfectly common in large parts of Europe. I also have a 3 months notice period, which is pretty much the standard here in Sweden for tech workers or other highly-qualified occupations. For very senior positions, 6 months isn't unheard of. It goes both ways of course, my employer is also bound by the 3 month period in case they want to fire me.
How does getting a new job work? You tell the new company that you have to give 3 months notice? That's some long term planning on the part of the hiring company.
Re: The absolute worst scenario happened
#100Earlier quoted context omitted.
These types of situations always wind up being the most fun I have at work and leave me feeling most fulfilled. I wish I could do that all day instead of going to standups.
A "firefighting" consultancy might be fun, no? And you would give people a lot of time off!
If anyone happens to know of a career path or company that does that sort of work I would be interested to hear about it. Bonus points if the pay is half decent.