Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The absolute worst scenario happened

reddit.com

Post reply on HN
Print view View original

271–280 of 566 posts

Re: The absolute worst scenario happened

#271

Post by undefined1 »

good idea in that thread:

If you don't have a backup of your zone files, a good way to quickly pick up the main domains being used would be to turn on logging for your DNS server and output into a log collector. That way you can quickly build queries on which IPs are asking for which FQDNs and start rebuilding a list of your lost zones and can maybe do some guess-work on which IPs those FQDNs were likely to resolve to.

I would expect irregular or extremely infrequent processes that use specific FQDNs will pop up from time to time as errors/failed processes and should prime IT teams on what to look out for.

https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...

Re: The absolute worst scenario happened

#272

Post by hackeraccount »

ansible wrote: post #61
rebuilder wrote: post #51

Earlier quoted context omitted.

From a reply the poster made in that thread: "So, we have backups on an encrypted drive, which we can't access (authentication is done via our custom ERP software, which is down as well - in hindsight, a very bad idea). And the backups don't work. We can rebuild the DNS servers from scratch, and the records are probably still safely stored somewhere inone of the 3 different databases our ERP software uses, but we don…

> Elsewhere, they say that two years ago, they found they were unable to restore from backups... So their business actually failed a couple years ago, and they have only noticed this month.

Basically Wile E. Coyote running in the air.

Re: The absolute worst scenario happened

#273

Post by pratik661 »

4e530344963049 wrote: post #90
paulz_ wrote: post #86

Earlier quoted context omitted.

These types of situations always wind up being the most fun I have at work and leave me feeling most fulfilled. I wish I could do that all day instead of going to standups.

A "firefighting" consultancy might be fun, no? And you would give people a lot of time off!

I always wondered this:

If you are an external consultant (with no prior knowledge of the company's systems/processes/hierarchies) brought in to "fight" a particular fire, how do quickly get ramped up to a level of knowledge where you can analyze the root cause of/recommend a fix for a particular "fire"?

Re: The absolute worst scenario happened

#274

Post by ballenf »

scruple wrote: post #260
NewLogic wrote: post #167

Earlier quoted context omitted.

The exact same thing is happening in Australia in regards to farmers and fruit picking labour. With international borders shut they are unable to exploit foreign workers and complain that young adults are refusing to relocate for $5 an hour hard labour jobs.

> complain that young adults are refusing to relocate for $5 an hour hard labour jobs. USD $5/hour is what I earned as a teenager working the fields by hand (picking corn, beans, and fruit for sale same-day at the farmers small stands throughout the area) in the early 1990s... That's insane. Is that USD $5 or AD?

I think people will one day demand higher pay for desk jobs over outdoor work. Looking back on my early life, the hard outdoor work was incredibly valuable to me. Desk jobs sometimes were soul-sucking, depression inducing. Not always and my dev work now is desk work that doesn't suck.

Now, outdoor work with a high injury rate is a different story.

Re: The absolute worst scenario happened

#275

Post by at_a_remove »

Accujack wrote: post #256
tw04 wrote: post #88

I actually don't see any issue that isn't simply a matter of money. Everything they list that's an issue seems to revolve around the fact that the employees with domain knowledge for the system have left the company. Reach out to them and hire them on as contractors. If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out…

It's slightly more complicated than that, depending on how much time has passed since the creators of the system left, but you're correct in general. Usually, since admitting that they were stupid to let all the knowledge of the system walk out the door requires more emotional and intellectual flexibility than managers who let this sort of thing happen possess, the biggest obstacle isn't even the money... it's gettin…

Yes. I had an ancient system where the holders of domain knowledge kept either dying or retiring, one by one. I was frantic, but nobody listened to me. The system began to fail on the day I announced my departure. I would later find that one of the previous architects had reached out to my manager to offer assistance, but this was never made known to us. It wouldn't have even cost anything.

Anything but pride, I guess.

Re: The absolute worst scenario happened

#276

Post by FireBeyond »

wing-_-nuts wrote: post #262
NewLogic wrote: post #167

Earlier quoted context omitted.

The exact same thing is happening in Australia in regards to farmers and fruit picking labour. With international borders shut they are unable to exploit foreign workers and complain that young adults are refusing to relocate for $5 an hour hard labour jobs.

Here in america farmers hire illegals to harvest crops, claiming that no american would do the work. This has always struck me as a really terrible excuse. I'm sorry, you can't simply break the law because you'd have to pay an american $20 / hr to do the work. Sure, it means that your tomatoes are more expensive, but also, now there's a hell of an incentive to apply automation to the problem yes? If you don't like th…

A couple of years ago, a serial offender food producer was raided, multiple times. 600+ undocumented workers were found, and all spoke of management knowing their status.

The workers were detained, deported, and so on.

After several raids, not so much as a misdemeanor was levied against a single person, nor a fine to the company in general.

Re: The absolute worst scenario happened

#277

Post by ihsw »

antihero wrote: post #208
WJW wrote: post #194

Earlier quoted context omitted.

...yes? The alternative to farmers is not harvesting their crops at all, since with the borders closed foreign labor is not available. If nobody wants to buy $10 doritos, that means the product can't exist in a market without cheap labor.

Well perhaps that's the problem of the market. That it can't particularly function to enrich some people without gross levels of exploitation.

Being paid $5/hour farming in Australia vs $0.50/hour assembling shoes in Malaysia. Wow, such exploitation.

"Exploitation" is a relative term, some people would rather be exploited in the countryside of a first-world nation rather than the countryside of their homeland.

Re: The absolute worst scenario happened

#278

Post by TeMPOraL »

michaelt wrote: post #226
TeMPOraL wrote: post #94

Earlier quoted context omitted.

If the system is old (going by other replies, introduced in a deal 20 years ago), maybe they'll be able to brute-force the encryption...

Encryption being 20 years old doesn't mean it's easy to break, given that AES is 23 years old :)

It doesn't, but in my experience, the ciphers and key sizes commonly used were often not future-proof :). This rests on the assumption that nobody updated the keys/encryption components in 20 years - which I think is quite possible with such internal enterprise deployment.

Re: The absolute worst scenario happened

#279

Post by Symbiote »

smnrchrds wrote: post #245
CydeWeys wrote: post #231

Earlier quoted context omitted.

People have to eat food. Food in aggregate has one of the most inelastic demand curves imaginable. If the price rises because the production costs increase, people will still buy food. They might buy different food, and you'll see a decrease in acreage given over to labor-intensive crops, but that's just the free market at work.

The thing is, if farmers in Australia increase wages and thus their product becomes more expensive, the product on supermarket shelves will not become more expensive. Rather, supermarkets will import food from another country. Food is, with few exceptions, a commodity. The producer does not set the price of the commodity, the market does. I am telling this as an engineer working in the Canada's oil and gas sector. Si…

The EU sets fairly high tariffs on food imports, since food security means we must maintain local food production.

Australia does not use this approach, and instead seems to have pushed other countries to eliminate their own tariffs for the TPP.

Re: The absolute worst scenario happened

#280

Post by digi59404 »

notyourday wrote: post #202
beckingz wrote: post #178

Earlier quoted context omitted.

hopefully you can skip most of those people and talk to the people who are in danger of being eliminated if your services are NOT engaged?

CEO: We are doing project X. We have engaged so and so to do it. You are to support this. : I'm uncomfortable sharing this information with the outside party. I will need to get the approvals for ... CEO: Yes, tell them to do this. ... Jack, head of IT in charge of backups: Ok, you can have this access. Consultant: Great. It says I'm not authorized to perform this operation. It is a blocker, could this be fixed so I…

This happens, and honestly it happens rarely.

When you’re in this role your goal and job is to build relationships and help people. That includes Jack. The job is getting done with or without Jacks help. So he can either jump onboard or he can be removed and someone else can be added to give access.

In the end people like Jack benefit nothing from being gatekeepers. Because the third time I have to ask for permissions to be changed - I’m asking for Jack to be removed from the task and someone else to be put in place.

Post reply on HN