Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The absolute worst scenario happened

reddit.com

Post reply on HN
Print view View original

191–200 of 566 posts

Re: The absolute worst scenario happened

#191

Post by drummojg »

capableweb wrote: post #60
flaxton wrote: post #55

Can’t you just pull the drive from the failed server, extract the DNS entries and build a new DNS server? It doesn’t sound like everything is broken, just that it is failing because the DNS server isn’t answering. Just install Linux on a new box and set up a DNS server on it. It could be done in a day I would expect. Or am I missing something?

It would work if they knew what the DNS entries were but seems they lost all DNS records and don't even know what records they have/had, so spinning up a new DNS server won't help as they don't know what it should serve. > All domain information was wiped out and records became null [...] Our records are wiped from all domain servers out there [...] We don't even know what domains we own, the listing was hosted in th…

That's part of what bothers me about this whole story--it says they were running BIND, which configs on text files for goodness sake. These critical records were tiny and could fit anywhere & transfer in the blink of an eye. That such a simple thing is buried under & dependent upon an entire complex and untested/maintained DR plan is mind-boggling.

Re: The absolute worst scenario happened

#192

Post by ACS_Solver »

hvidgaard wrote: post #162
ACS_Solver wrote: post #105

Earlier quoted context omitted.

In my EU-centric worldview, the US system of at-will employment, and of two-week notices as the standard courtesy, is crazy. Three months is the typical notice period for tech workers and other highly-skilled jobs. One month is the legal minimum in Sweden, and certainly several other countries. Of course it goes both ways, in fact it's even more difficult for the employer to get rid of an employee. An employee doesn'…

In reality, you can walk out of any job in EU as well. Unless you have some specific clauses in your contract, the worst they can do is simply not pay you anything. There may come a lawsuit after, but I doubt it. You can also get a doctors note and go on sick leave if you feel some serious stress on the job.

It's true that they probably won't sue you for breach of contract, though they can, but if you just walk out of a job, you're at the risk of ruining your professional reputation, arguably the most precious resource. There aren't many other things that would be as much of a red flag when hiring as an employee that just decided to walk out of a job.

Of course if you have good reasons to stop working, there are good options. If it's stress or similar mental health issues, you can resign and spend your notice period on sick leave. If your employer assaults you or just stops paying, or otherwise grossly neglects their duties as an employer, you're legally entitled to quit immediately.

Re: The absolute worst scenario happened

#193

Post by pw201 »

arethuza wrote: post #68
PeterisP wrote: post #50

Earlier quoted context omitted.

What would be the practical consequences to the employee if they break that notice period in UK, i.e. stop coming to work (or, especially now with Covid, simply stop doing the remote work) sooner than these 3 months have passed?

I suspect you could be sued by your employer. Mind you I've never heard of anyone being sued in that way - can't imagine it would be worth it for the employer unless they could demonstrate a significant financial loss.

I investigated this once (while I was sitting around doing not much during a 3 month notice period, funnily enough). It's as you say, the breach of a contract is something where the employer would have to sue you for damages, and if you're sitting around not doing much, it's going to be hard to argue that losing you has caused them harm, much less enough to make it worth suing over.

I didn't break the contract anyway out of some combination of feeling like it was breaking a promise and the worry that Cambridge is a small town.

Re: The absolute worst scenario happened

#194

Post by WJW »

kolbe wrote: post #186
NewLogic wrote: post #167

Earlier quoted context omitted.

The exact same thing is happening in Australia in regards to farmers and fruit picking labour. With international borders shut they are unable to exploit foreign workers and complain that young adults are refusing to relocate for $5 an hour hard labour jobs.

Farming is low margin. Farmers don’t get to set the price of their products, and they aren’t handed billions in capital to burn through or 1000x multiples on their profit in the public markets. When most businesses say they can’t hire someone and the reason is because they don’t pay enough, the underlying cause of their low pay is usually because that’s what the company can afford. Sure, farmers could pay $10/hr, but…

...yes? The alternative to farmers is not harvesting their crops at all, since with the borders closed foreign labor is not available.

If nobody wants to buy $10 doritos, that means the product can't exist in a market without cheap labor.

Re: The absolute worst scenario happened

#195

Post by carschno »

amachefe wrote: post #98
batch12 wrote: post #80

Earlier quoted context omitted.

One thing the situation does show is a lack of proper care for their most critical assets. When the dust settles, there should be hard questions for several people. One being-- when was the last time backups were tested..

The answer to the question is there >We can't access the backups. Even if we could, we noticed two years ago when trying a restore from the backups, that it doesn't work. Booting the restored server leads to a kernel panic we couldn't figure out. Management said we don't have enough money to fix any of this.

I have no reason to doubt that this is factual. However, I have serious doubt that management will acknowledge they made that decision.

Typically, this kind of decisions are made more or less implicitly during long meetings about priorities, often without any minutes. Managers will presumably try to hold (senior) engineers accountable as they had not made clear enough how important that was.

I obviously don't know who is right here.

Re: The absolute worst scenario happened

#196

Post by ghaff »

garmaine wrote: post #183
tkgally wrote: post #159

Earlier quoted context omitted.

I remember reading many years ago that employees in the U.S. have the right to quit at any time because being penalized for quitting would make the employment “involuntary servitude,” which is forbidden by the Thirteenth Amendment to the U.S. Constitution. I can’t find any confirmation of that explanation on the web now, though.

This is correct. If you don't have the right to quit at any time, you are legally a slave / indentured servant. And that is very, very illegal.

The military is sort of a special case however. (And, of course, private employment contracts can make it financially onerous to break.)

Re: The absolute worst scenario happened

#197

Post by capableweb »

drummojg wrote: post #191
capableweb wrote: post #60

Earlier quoted context omitted.

It would work if they knew what the DNS entries were but seems they lost all DNS records and don't even know what records they have/had, so spinning up a new DNS server won't help as they don't know what it should serve. > All domain information was wiped out and records became null [...] Our records are wiped from all domain servers out there [...] We don't even know what domains we own, the listing was hosted in th…

That's part of what bothers me about this whole story--it says they were running BIND, which configs on text files for goodness sake. These critical records were tiny and could fit anywhere & transfer in the blink of an eye. That such a simple thing is buried under & dependent upon an entire complex and untested/maintained DR plan is mind-boggling.

> That such a simple thing is buried under & dependent upon an entire complex and untested/maintained DR plan is mind-boggling

I find this to pretty common in many setups where the engineers don't focus on simplicity and removing layers of abstraction. If the workforce is young and inexperienced, over-engineering tends to happen everywhere and you end with situations like this.

I have seen worse in my years in the industry, that's for sure.

Re: The absolute worst scenario happened

#199

Post by jusssi »

technics256 wrote: post #37
IgorPartola wrote: post #4

What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?

Depends on local law. In this case the user is in Finland under Finnish labor laws.

I just checked for curiosity. The employer can sue for damages they'll have to show.

But there's some additional factors here. If the employer misses salary payments, then they're in breach of that employment contract, and unless they can't fix the situation really soon, that's grounds for immediate termination.

Also in Finland, salaries earned in a normal employment are backed by the state for up to 3 months of missed payments (the state will then try to collect from the employer). So it's relatively safe for an employee to stick with a sinking company. Just get that CV polished and start warming up the network.

Re: The absolute worst scenario happened

#200

Post by TheRealDunkirk »

notyourday wrote: post #123
viraptor wrote: post #108

Earlier quoted context omitted.

There's the possibility that those employees are happily retired and don't care. Or they're just not available in many ways. I know of a hospital which is still refusing to migrate off of a system which is not supported for a decade. Pay more money? 3 people worked on it: Adam had a stroke, Bob retired with enough money, Charlie left the country.

I bet if the hospital is to pay Bob/Charlie $1M for 6 months, post tax to fix it they would gladly do it. Note: this does not mean "We will pay you up to a million dollars, Net 90, after an invoice is approved and signed off by fifteen people". This means "We will wire a deposit of $250,000 today. The rest will be wired every month in 24 hours after an invoice is issued" The reality is company execs like to talk abou…

This modern-day feudalism with corporations is really the root of the problem. Managers "patch" broken IT systems with lots of manual labor to actively avoid pointing out how some other manager's system is borked. They doom the company to spend hundreds or even thousands of man hours a year to work around these processes, refusing to allow it to be fixed, keeping the "power" and influence of the broken system with some other manager. They do this hoping to curry favor for the sake of their career prospects.

After 25 years, I'm an expert at making small, efficient software tools to fix local IT workflows. God knows, in the Fortune 250's I've worked at, there are endless opportunities. And, while I've written many solutions, I've seen this political game of thrones thwart my efforts many times. This is how you get the kind of 20-year legacy mess described in the original post: People sacrificing long-term success of the company for short-term success in their careers.

I guess there are 2 kinds of people. Those that think this is just how the world works, and find this to be a perfectly-cromulent way of getting through it, and those, like me, who find it odious. Luckily, I now work for people like me. We do what we can.

Post reply on HN