Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The absolute worst scenario happened

reddit.com

Post reply on HN
Print view View original

251–260 of 566 posts

Re: The absolute worst scenario happened

#251

Post by staticman2 »

tkgally wrote: post #159
ACS_Solver wrote: post #105

Earlier quoted context omitted.

In my EU-centric worldview, the US system of at-will employment, and of two-week notices as the standard courtesy, is crazy. Three months is the typical notice period for tech workers and other highly-skilled jobs. One month is the legal minimum in Sweden, and certainly several other countries. Of course it goes both ways, in fact it's even more difficult for the employer to get rid of an employee. An employee doesn'…

I remember reading many years ago that employees in the U.S. have the right to quit at any time because being penalized for quitting would make the employment “involuntary servitude,” which is forbidden by the Thirteenth Amendment to the U.S. Constitution. I can’t find any confirmation of that explanation on the web now, though.

You can quit but they can sue you for breach of contract.

I think what you are thinking about is the court almost certainly won't order you to go back to work after you quit.

Re: The absolute worst scenario happened

#252

Post by JKCalhoun »

tw04 wrote: post #88

I actually don't see any issue that isn't simply a matter of money. Everything they list that's an issue seems to revolve around the fact that the employees with domain knowledge for the system have left the company. Reach out to them and hire them on as contractors. If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out…

Is there another industry where a few employees with "domain knowledge" can sink a company?

Asking because, while, yes, throwing money at the problem might be a solution, I think the obvious bigger issue is how does one allow themselves to get into this situation?

Or rather, how does one avoid getting into this situation. Because it seems to me that the software industry of late has become so mired in esoteric layers of abstraction that this feels less like an outlier scenario for a mismanaged company but more like an inevitable one for many otherwise well managed companies in the industry.

Re: The absolute worst scenario happened

#253

Post by ansible »

arethuza wrote: post #76
WJW wrote: post #17

Earlier quoted context omitted.

It does say in the original post that: > Our backups are fucked. So I assume that they have already tried and they didn't work.

Early in my career (>30 years) ago I nuked a companies salary database (a missing $ in a shell script moved everything to the file i) No problem they said - we have backups. They had three tapes. First tape failed. Second tape failed. Third tape worked. In retrospect that was a useful, if rather stressful, lesson.

It is also helpful to have at least two tape drives, and check the backup tapes against both.

Early in my career, we had a storm that caused an electrical surge. In spite of the power protection, it damaged the tape drive on the main server (unknown to us at the time). The entire rest of the system (CPU, RAM, hard drives (RLL!)) were fine.

Listening to the backups being run, I noticed it sounded a little different. Eventually I determined that the tape drive (old QIC, maybe 40MB capacity?) worked, but it would not switch tracks when reading / writing. So the backup ran, but it just overwrite the same track rather than write to subsequent tracks.

A short backup / restore wouldn't have caught the problem, it was necessary to try to read the whole backup to see. This wasn't caught for a while, so all the backups were mostly useless.

Re: The absolute worst scenario happened

#254

Post by phkahler »

asdfman123 wrote: post #232
drclau wrote: post #132

Earlier quoted context omitted.

Not really related to the subject of the article, but your comment reminds me of employers in Romania complaining they can't find people to hire, while they offer small salaries and people just leave the country for better paying jobs abroad. The solution to their problem is really simple: make better offers.

Well, the problem is this: it might not make economic sense for those specific companies to hire people at a higher rate. Companies aren't going to say "I guess we're inefficient and should just die." They're going to fight to survive and thrive, which is what you're seeing. Then again, some companies ARE just cheap. You never know.

I was at a hotel with crappy service and asked about the problems with house keeping. The manager told me everyone in town (Traverse City MI) was trying to bring in foreigners to do housekeeping because they couldn't get local labor.

They said the standard is 30 minutes per room per day. I figure at a very generous $50 per hour (100k / year) it should run $25 to have my room taken care of. Now that doesnt cover the shared costs - doing laundry, cleaning the pool, front desk, etc... but my room was something like $150 per night. How the fuck can they not find people to clean rooms?

Answer: they're too cheap to pay a decent wage. But why? My suspicion is that all these places are actually leveraged far too much and paying rent/interest/dividends so much they can barely operate.

I suspect low interest rates and other incentives to "stimulate" economies are actually causing these problems.

Re: The absolute worst scenario happened

#255

Post by ISL »

JKCalhoun wrote: post #252
tw04 wrote: post #88

I actually don't see any issue that isn't simply a matter of money. Everything they list that's an issue seems to revolve around the fact that the employees with domain knowledge for the system have left the company. Reach out to them and hire them on as contractors. If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out…

Is there another industry where a few employees with "domain knowledge" can sink a company? Asking because, while, yes, throwing money at the problem might be a solution, I think the obvious bigger issue is how does one allow themselves to get into this situation? Or rather, how does one avoid getting into this situation. Because it seems to me that the software industry of late has become so mired in esoteric layers…

> Or rather, how does one avoid getting into this situation.

Documentation?

Re: The absolute worst scenario happened

#256

Post by Accujack »

tw04 wrote: post #88

I actually don't see any issue that isn't simply a matter of money. Everything they list that's an issue seems to revolve around the fact that the employees with domain knowledge for the system have left the company. Reach out to them and hire them on as contractors. If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out…

It's slightly more complicated than that, depending on how much time has passed since the creators of the system left, but you're correct in general.

Usually, since admitting that they were stupid to let all the knowledge of the system walk out the door requires more emotional and intellectual flexibility than managers who let this sort of thing happen possess, the biggest obstacle isn't even the money... it's getting the managers to admit they're idiots.

Re: The absolute worst scenario happened

#257

Post by lmkg »

etripe wrote: post #230
TheRealDunkirk wrote: post #153

One of the comments in the thread said, simply, "Prepare 3 envelopes." Being a 25-year sysadmin, immersed in thinking about the problem, it caught me totally off-guard. I don't think I've ever seen a more perfect response in a Reddit thread.

Which 3 envelopes would those be?

It's an old story.

New executive starts the job. On his desk are three envelopes, left by his predecessor, with a post it note saying "When shit hits the fan, and you don't know what to, open one of the envelopes."

Yadda yadda, exposition, things are normal and exec is successful for a year or two and then things go pear-shaped. Completely out of options, he opens the envelope, finds a letter which says "Blame me." He blames his predecessor for leaving the state of things in such a poor shape that this misfortunate was unavoidable, everything blows over, his success continues.

Blah blah, narration. Another shitstorm. He opens the second envelope, finds a letter which says "fire all the managers." He promises a significant shake-up, says he has the leadership to turn this around, purges the old guard. Things go well.

Hurly-burly story, and so forth. Catastrophe strikes. Having blamed his predecessor and fired everyone and installed his own managers, he has no one else to turn to for casing the blame. He opens the last envelope, hoping to find salvation.

"Write three letters."

Re: The absolute worst scenario happened

#258

Post by abruzzi »

jiggawatts wrote: post #133
mysterydip wrote: post #113

Earlier quoted context omitted.

honestly, "password" would be my first guess.

Oh come now, there are password complexity policies in place in any modern software! Try: password123.

it doesn't sound like modern software. At my current employer (to remain unnamed) when I started there were a bunch of MS-SQL servers with an SA password of (i.e. no password). It took a long time to fix that because we had hundreds of local installs of the client app that had SA/ in the app config file (yes, they were setup to access as system administrator), so fixing it meant reconfiguring hundreds of local installs. (at the time we didn't have a mechanism for automating it.) We still have "enterprise" software that does no password rules, and would happily allow a blank password.

Re: The absolute worst scenario happened

#259

Post by pydry »

danjac wrote: post #151
scandox wrote: post #35

How people handle this level of disaster is always very revealing. In every case I've ever experienced someone kept their head and a solution was found. Usually there was reputational damage but surprisingly most times the actual business bounced back. But it was always the actions of someone in particular in those first hours that made the difference. Thinking you're fucked is unhelpful, because you start talking ab…

Looking through the comments, it appears concerns were raised for years and management refused to allow the necessary changes to save money. While you can and should be the cool head in the room, it's a route to burnout if you end up being the hero who has to keep saving a dysfunctional organization and management from the consequences of its own failures. By all means do your best fix the immediate problem but get t…

>it's a route to burnout if you end up being the hero who has to keep saving a dysfunctional organization

The same dysfunction that led to disaster after disaster is likely to lead to very little reward (if any) for actually being the hero, also.

Re: The absolute worst scenario happened

#260

Post by scruple »

NewLogic wrote: post #167
drclau wrote: post #132

Earlier quoted context omitted.

Not really related to the subject of the article, but your comment reminds me of employers in Romania complaining they can't find people to hire, while they offer small salaries and people just leave the country for better paying jobs abroad. The solution to their problem is really simple: make better offers.

The exact same thing is happening in Australia in regards to farmers and fruit picking labour. With international borders shut they are unable to exploit foreign workers and complain that young adults are refusing to relocate for $5 an hour hard labour jobs.

> complain that young adults are refusing to relocate for $5 an hour hard labour jobs.

USD $5/hour is what I earned as a teenager working the fields by hand (picking corn, beans, and fruit for sale same-day at the farmers small stands throughout the area) in the early 1990s... That's insane. Is that USD $5 or AD?

Post reply on HN