Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The absolute worst scenario happened

reddit.com

Post reply on HN
Print view View original

451–460 of 566 posts

Re: The absolute worst scenario happened

#451

Post by kodah »

I'm a systems engineer and a software engineer. A couple tips that are relevant to this story:

1. Applications do not need DNS servers, ever. Hosting platforms do. The way I read this is that it's some odd form of split horizon where one DNS server seems to replicate to larger DNS servers, yet the single server is what carries most of the critical information.

2. State should be transferred from where you gather input to where it becomes actionable by an eventually consistent process. ie: If you store your customer CNAMEs in ERP, then something like webhooks to a service that maintains the state in machine readable form should verify, accept/deny, and do something with that change (like update DNS).

3. Customer systems and core/critical systems should be separate. The fact that critical contact systems for employees were not separate from

4. Have a DR plan and artificially enact it at cadence, doing so even in a sterile environment is better than nothing. Successful DR's in production are best. If the first time you live test your DR strategy is when all is lost, then all is most likely lost and at best you have a lot of hours of work ahead of you.

5. The holy grail is replayable audit frameworks. This could mean capturing diffs but it can be as simple as logging the current state of a given $thing and sending it to an off-local system.

Re: The absolute worst scenario happened

#452

Post by mixedCase »

jandrese wrote: post #327
greeneggs wrote: post #286

Earlier quoted context omitted.

Of course it isn't that simple. If your farm's tomatoes are more expensive than your neighbors' tomatoes, then nobody will buy them. You'll quickly go bankrupt. It is also difficult because of the two tiers of labor. A farmer will try increasing wages by $1/hour, and still struggle to find enough workers. Wages would probably have to be doubled before new workers—Americans—would start to enter the industry.

This is the reason tariffs exist. You levy a fee that makes the imported food cost as much as food picked using domestic labor.

Or, you know, you let unprofitable endeavors be substituted by profitable ones, and you just import food from different places where it is profitable. Just make sure not to wage war with them all. That's always an alternative to subsidies and tariffs, but it's not as politically useful.

Re: The absolute worst scenario happened

#453

Post by AdmiralGinge »

NewLogic wrote: post #167
drclau wrote: post #132

Earlier quoted context omitted.

Not really related to the subject of the article, but your comment reminds me of employers in Romania complaining they can't find people to hire, while they offer small salaries and people just leave the country for better paying jobs abroad. The solution to their problem is really simple: make better offers.

The exact same thing is happening in Australia in regards to farmers and fruit picking labour. With international borders shut they are unable to exploit foreign workers and complain that young adults are refusing to relocate for $5 an hour hard labour jobs.

Sounds like exactly the same problem British farmers are having post-Brexit with things like fruit-picking, they're scratching their heads wondering why British workers aren't keen on having half their wages taken for "rent" to live six people to a crappy caravan on-site and do backbreaking labour day in day out. Their only choices are to improve their wages, conditions, both, or go bankrupt.

Re: The absolute worst scenario happened

#454

Post by pg_bot »

rjmunro wrote: post #333
colinmhayes wrote: post #313

Earlier quoted context omitted.

If farmers paid enough to get Americans to do seasonal farm labor their produce would cost twice as much as central/south American produce and they would immediately go out of business as everyone buys cheap imported fruit.

You can fix this with import duties. If the country you are importing from doesn't have good labor laws, you charge them 100% extra.

Now everyone is poorer.

Re: The absolute worst scenario happened

#455

Post by anyfoo »

alistairSH wrote: post #118
ACS_Solver wrote: post #105

Earlier quoted context omitted.

In my EU-centric worldview, the US system of at-will employment, and of two-week notices as the standard courtesy, is crazy. Three months is the typical notice period for tech workers and other highly-skilled jobs. One month is the legal minimum in Sweden, and certainly several other countries. Of course it goes both ways, in fact it's even more difficult for the employer to get rid of an employee. An employee doesn'…

In my EU-centric worldview, the US system of at-will employment, and of two-week notices as the standard courtesy, is crazy. Oh, I agree. But 3 months seems crazy to me. If I have an open position and it isn't filled in ~4-6 weeks, it often gets given to some other team (on the basis that if I really needed that position filled, I'd do so). I can't plan hiring over that timeframe. And when hiring employees in India,…

I had a "to the next quarter" period in my last job, and when moving to the new job the deadline was missed, making the contract valid for a whole quarter.

It wasn't a problem. I was on good terms with the previous company (I'm still getting invited to their anniversary parties and stuff like that), so there was no issue in working something out.

Also, remember that ~6 weeks of vacation per year is common outside the US, so often it boils down to taking your remaining vacation. (Though I think you need to be careful with taxes and other regulations if the contracts overlap.)

Re: The absolute worst scenario happened

#456

Post by OneLeggedCat »

throwthescene wrote: post #305
tw04 wrote: post #88

I actually don't see any issue that isn't simply a matter of money. Everything they list that's an issue seems to revolve around the fact that the employees with domain knowledge for the system have left the company. Reach out to them and hire them on as contractors. If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out…

Throwaway purely so that I can comment. I've had this happen before. Years ago I left a rather awful company - I left on great terms with my boss, though I hated the company. I was the last person who knew how to release X product. While I documented everything, after I left, the wiki lost its mind, backups apparently weren't a thing, and the director of IT moved to Bermuda. Fast forward 6 months.. a contractual obli…

This is the most beautiful thing I've read today. Thank you.

Re: The absolute worst scenario happened

#457

Post by pro_zac »

Eremotherium wrote: post #166
driton wrote: post #114

Earlier quoted context omitted.

Sounds very interesting, but the story seems to be behind a paywall. I found some other articles[0][1] related to the incident, however none of them seem to mention the flying of a hard drive across continents. 0: https://www.zdnet.com/article/ransomware-the-key-lesson-maer... 1: https://portswigger.net/daily-swig/when-the-screens-went-bla...

Or if you want the story as a podcast I can wholeheartedly recommend this episode (and the podcast in its entirety) of Darknet Diaries: https://darknetdiaries.com/episode/54/

Seconded! Great story and amazing podcast.

Re: The absolute worst scenario happened

#458

Post by OneLeggedCat »

ycombinete wrote: post #322
tofuahdude wrote: post #309

Earlier quoted context omitted.

I love this with all my heart. You did exactly the right thing at every step.

Funnily enough it made me sick to my stomach. I manage a small company and the idea of being extorted in such a blasé and psychopathic fashion terrifies me to even think about.

Hopefully you wouldn't mismanage your own company and treat your employees like shit. Probably you are a better person than that, and can avoid this ever happening to you.

Re: The absolute worst scenario happened

#459

Post by vagrantJin »

ycombinete wrote: post #322
tofuahdude wrote: post #309

Earlier quoted context omitted.

I love this with all my heart. You did exactly the right thing at every step.

Funnily enough it made me sick to my stomach. I manage a small company and the idea of being extorted in such a blasé and psychopathic fashion terrifies me to even think about.

I see and understand your sentiment but, in my rather insignificant few years in the workforce - carrots, dipped in vanilla chocolate and honey, generally tend to work better than the stick.

The company could have apologised, and do a lot more ego stroking before asking a favour which they can pay for and maybe throw in an all expenses paid weekend with the SO as a thank you. Not only will you get goodwill but future problems are going to be significantly cheaper to solve. Even if you have a bad break up - as a business owner you should understand the value of mending bridges. It's cheaper than losing sleep at night and losing your steak dinner to a bad stomach.

Re: The absolute worst scenario happened

#460

Post by hinkley »

blhack wrote: post #375

It's dark, but I kindof love situations like this (although luckily I haven't had something like this happen to me in digital space since I was a teenager). Some places I would look: 1) Are you SURE that everything is actually gone? Is there a logfile, a billing trail, sent emails trail from your email provider, something like that? If it was me I'd be looking for a secondary system (like alerts, logfiles, billing, e…

Start connecting with everyone on LinkedIn for one...
Post reply on HN