Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The absolute worst scenario happened

reddit.com

Post reply on HN
Print view View original

371–380 of 566 posts

Re: The absolute worst scenario happened

#371

Post by kstrauser »

throwthescene wrote: post #305
tw04 wrote: post #88

I actually don't see any issue that isn't simply a matter of money. Everything they list that's an issue seems to revolve around the fact that the employees with domain knowledge for the system have left the company. Reach out to them and hire them on as contractors. If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out…

Throwaway purely so that I can comment. I've had this happen before. Years ago I left a rather awful company - I left on great terms with my boss, though I hated the company. I was the last person who knew how to release X product. While I documented everything, after I left, the wiki lost its mind, backups apparently weren't a thing, and the director of IT moved to Bermuda. Fast forward 6 months.. a contractual obli…

Hell. Yes. That's absolutely beautiful, and ethically completely correct.

I have bosses at previous companies that I'd help for free today because they were good to me and we parted on friendly terms. One boss helped me move my house once, and I'm not about to charge him to fix some random issue.

I have other bosses whom I'd require a cleared wire transfer of money before I'd lift a pinky to help them, because they were not nice people and I don't want to associate with them or assist them with anything. If someone in this camp approached me, I'd have a story just like yours afterward.

Re: The absolute worst scenario happened

#372

Post by andyv »

rolobio wrote: post #343
throwthescene wrote: post #305

Earlier quoted context omitted.

Throwaway purely so that I can comment. I've had this happen before. Years ago I left a rather awful company - I left on great terms with my boss, though I hated the company. I was the last person who knew how to release X product. While I documented everything, after I left, the wiki lost its mind, backups apparently weren't a thing, and the director of IT moved to Bermuda. Fast forward 6 months.. a contractual obli…

Why a check with your lawyers? Why not just deposit into your personal account?

With the check held in escrow, it makes cheating harder.

The contractor knows that the money has been paid by the company and won't be withheld after the job is done.

The company knows that the contractor doesn't have the money until the job is done.

Re: The absolute worst scenario happened

#373

Post by heterodoxxed »

ycombinete wrote: post #322
tofuahdude wrote: post #309

Earlier quoted context omitted.

I love this with all my heart. You did exactly the right thing at every step.

Funnily enough it made me sick to my stomach. I manage a small company and the idea of being extorted in such a blasé and psychopathic fashion terrifies me to even think about.

Have you tried learning to code?

Re: The absolute worst scenario happened

#374

Post by HarryHirsch »

aaron-santos wrote: post #282
FireBeyond wrote: post #276

Earlier quoted context omitted.

A couple of years ago, a serial offender food producer was raided, multiple times. 600+ undocumented workers were found, and all spoke of management knowing their status. The workers were detained, deported, and so on. After several raids, not so much as a misdemeanor was levied against a single person, nor a fine to the company in general.

Kind of makes me think, what's the point when these laws aren't actually enforced?

"For my friends, everything. For my enemies, the law." Getúlio Vargas had it all worked out.

Re: The absolute worst scenario happened

#375

Post by blhack »

It's dark, but I kindof love situations like this (although luckily I haven't had something like this happen to me in digital space since I was a teenager). Some places I would look:

1) Are you SURE that everything is actually gone? Is there a logfile, a billing trail, sent emails trail from your email provider, something like that? If it was me I'd be looking for a secondary system (like alerts, logfiles, billing, etc.) that had some of this information stored in it as secondary function, and writing some scripts to parse those things and start rebuilding what can be rebuilt.

2) You said you lost all the emails and SMS of your team. Do you have an HR department? Surely they have this information. Your team is likely going to start trying to send emails to each other about this. Can you dump logs off of your mail server to rebuild at least the naming schema? For phone numbers: can you try an old school phone tree style? Start with somebody you know, and ask them for everybody that THEY know and so on.

3) Was there a staging environment that might have had some of the data you're missing on it?

4) Are you really really sure that everything was wiped out? Is it possible that your ERP system is broken (lost its DB connection or something?) and that caused a cascading failure that makes it LOOK like everything is gone?

5) Can you get at the BIND server somehow? Are there logfiles there?

I'd love to know more specifics about what "wiped out" means in this case. It seems unlikely that it's properly "wiped out" unless something malicious happened, and even then I think you could rebuild a lot of this from sortof "secondary" data.

Think like a hacker, but your system is your own target. How can you steal the data you're looking for from places it isn't really "supposed" to be?

Re: The absolute worst scenario happened

#376

Post by Ekaros »

jusssi wrote: post #199
technics256 wrote: post #37

Earlier quoted context omitted.

Depends on local law. In this case the user is in Finland under Finnish labor laws.

I just checked for curiosity. The employer can sue for damages they'll have to show. But there's some additional factors here. If the employer misses salary payments, then they're in breach of that employment contract, and unless they can't fix the situation really soon, that's grounds for immediate termination. Also in Finland, salaries earned in a normal employment are backed by the state for up to 3 months of miss…

It seems to me that they can sue for 3 months salary. Or the amount of days employee does not work. However in this situation I would imagine they have more immediate concerns so might not get around even to that.

Re: The absolute worst scenario happened

#377

Post by notabee »

TheRealDunkirk wrote: post #200
notyourday wrote: post #123

Earlier quoted context omitted.

I bet if the hospital is to pay Bob/Charlie $1M for 6 months, post tax to fix it they would gladly do it. Note: this does not mean "We will pay you up to a million dollars, Net 90, after an invoice is approved and signed off by fifteen people". This means "We will wire a deposit of $250,000 today. The rest will be wired every month in 24 hours after an invoice is issued" The reality is company execs like to talk abou…

This modern-day feudalism with corporations is really the root of the problem. Managers "patch" broken IT systems with lots of manual labor to actively avoid pointing out how some other manager's system is borked. They doom the company to spend hundreds or even thousands of man hours a year to work around these processes, refusing to allow it to be fixed, keeping the "power" and influence of the broken system with so…

This applies to similar problems in government as well. Some people care, some just kick the can to whoever is unfortunate enough to inherit the problem in 5-10 years. If you get a few of the latter people in political power, often the best thought out solutions are offered many times and denied because of internal political wrangling.

Re: The absolute worst scenario happened

#378

Post by jbroson »

reaperducer wrote: post #335
wing-_-nuts wrote: post #262

Earlier quoted context omitted.

Here in america farmers hire illegals to harvest crops, claiming that no american would do the work. This has always struck me as a really terrible excuse. I'm sorry, you can't simply break the law because you'd have to pay an american $20 / hr to do the work. Sure, it means that your tomatoes are more expensive, but also, now there's a hell of an incentive to apply automation to the problem yes? If you don't like th…

Here in america farmers hire illegals to harvest crops, claiming that no american would do the work This is a political talking point that has been repeated so often that people believe it is true. During the recession of 2008, there were stores on local TV news in many places showing unemployed Americans waiting in long lines for the opportunity do do those jobs that the politicians kept saying no American wanted to…

My somewhat educated guess is that these farmers in the PNW don't share the same politics as farmers in the rural south or midwest.

That and Republicans have grown way more hostile to immigration since 2010, so it's an interesting anecdote but not sure how well it translates across the US, where polling shows farmers are overwhelmingly Republican who tend also tend to hate immigration legal or not.

Re: The absolute worst scenario happened

#379

Post by blhack »

4e530344963049 wrote: post #67
scandox wrote: post #35

How people handle this level of disaster is always very revealing. In every case I've ever experienced someone kept their head and a solution was found. Usually there was reputational damage but surprisingly most times the actual business bounced back. But it was always the actions of someone in particular in those first hours that made the difference. Thinking you're fucked is unhelpful, because you start talking ab…

If you have been that person before, it can be fun and exciting to think up solutions and implement them under pressure. Definitely a real test in your ability to concentrate. But don't expect the credit you deserve for it, and don't even expect others to even bother to put in any effort towards fixing things. Also, make sure everyone knows you are going on vacation once you resolve things.

This is basically the reason I got out of doing IT and into development.

"If you do things well enough, nobody will think you did anything at all."

Re: The absolute worst scenario happened

#380

Post by beckingz »

notyourday wrote: post #202
beckingz wrote: post #178

Earlier quoted context omitted.

hopefully you can skip most of those people and talk to the people who are in danger of being eliminated if your services are NOT engaged?

CEO: We are doing project X. We have engaged so and so to do it. You are to support this. : I'm uncomfortable sharing this information with the outside party. I will need to get the approvals for ... CEO: Yes, tell them to do this. ... Jack, head of IT in charge of backups: Ok, you can have this access. Consultant: Great. It says I'm not authorized to perform this operation. It is a blocker, could this be fixed so I…

Oh.
Post reply on HN