Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The absolute worst scenario happened

reddit.com

Post reply on HN
Print view View original

461–470 of 566 posts

Re: The absolute worst scenario happened

#461

Post by toyg »

m-p-3 wrote: post #435
throwthescene wrote: post #305

Earlier quoted context omitted.

Throwaway purely so that I can comment. I've had this happen before. Years ago I left a rather awful company - I left on great terms with my boss, though I hated the company. I was the last person who knew how to release X product. While I documented everything, after I left, the wiki lost its mind, backups apparently weren't a thing, and the director of IT moved to Bermuda. Fast forward 6 months.. a contractual obli…

Damn, I'd pay a good part of my mortgage in 4 days with that. You played your cards well.

“A good part”? Lol, I’d pay it all and have enough spare to buy a midrange car...

Re: The absolute worst scenario happened

#462

Post by hinkley »

sorokod wrote: post #27

"Every person who knew how the custom DNS system worked has left the company years ago." Keeping the people who know how the system works is really the last line of defense. From, the point where such last person left without any compensating mechanism in place, the clock started ticking

Somewhere right now there is a disgruntled former employee drinking a glass of Scotch and smiling.

Re: The absolute worst scenario happened

#463

Post by hinkley »

gillesjacobs wrote: post #26

The managed service provider company built a custom DNS inside an ERP without backups or bootstrap strategy in place. House-of-cards core infrastructure is a sign of negligence and OP should have reconsidered working there a long time ago.

Division of responsibility can make this sort of thing very difficult to know.

If you aren’t running drills then transparency may not be in everybody’s self interest and someone somewhere is covering up for the fact that they only half know what they’re doing.

Re: The absolute worst scenario happened

#464

Post by smnrchrds »

zentiggr wrote: post #436
smnrchrds wrote: post #245

Earlier quoted context omitted.

The thing is, if farmers in Australia increase wages and thus their product becomes more expensive, the product on supermarket shelves will not become more expensive. Rather, supermarkets will import food from another country. Food is, with few exceptions, a commodity. The producer does not set the price of the commodity, the market does. I am telling this as an engineer working in the Canada's oil and gas sector. Si…

So.... reduce production, and supply/demand curves should get you back financially on track? Of course, you might get pilloried for the artificial scarcity, but hey... nobody ever likes the capitalist :)

We reduce production, US, Russia, and Saudi Arabia would be more than happy to fill the gap. Actually Saudi Arabia prefers prices that are lower than the strike point for non-traditional oil, so it can get rid of competition from likes of Canada. How should we convince Russia and Saudi Arabia to take a hit to their exports in order to help us out, when we cannot even convince our biggest ally, the US?

Re: The absolute worst scenario happened

#467

Post by hinkley »

ChrisMarshallNY wrote: post #30

This is the snow on the tip of the iceberg. I'm sure there's plenty more where this came from. There's so many things wrong with the scenario described, that I can't even begin to talk about it. My experience is that folks never want to talk about backup or DR, as they are expensive, and the idea is that they should never be used. No one ever has a problem with insurance premiums, though...

I keep wondering if there’s an underwriter angle here where you offer insurance and audit the company to set the premiums based on how broken their IT situation is.

Re: The absolute worst scenario happened

#468

Post by perlgeek »

xchaotic wrote: post #72
benlumen wrote: post #44

Earlier quoted context omitted.

Thanks. Reddit is such a good example of how "progress" hasn't really been progress in web development. The new front-end framework site is so much slower and more buggy.

wow, it was so much better before. What the heck? Who made the decision to switch?

The old design was optimized for usability, the new is optimized for ad revenue and/or for pushing people on the app.

Re: The absolute worst scenario happened

#469

Post by chmod600 »

There would be a lot of value in independent organizations that can evaluate engineering practices. Non-technical businesses could just ask if you have cert XYZ before paying you.

It wouldn't have to get into the weeds too much. I mean really basic stuff like finding documents and following them to restore a backup or release a new version or fix a simple bug injected in the system.

Re: The absolute worst scenario happened

#470

Post by Symbiote »

benlivengood wrote: post #405
kwhitefoot wrote: post #143

Earlier quoted context omitted.

> Management said we don't have enough money to fix any of this. It's often even worse. Many years ago I was responsible for backup a DEC Unix system. Asked for funds to conduct a disaster recover exercise to find out if our backups worked and was turned down even before we had tried to work out how much it would cost. I often wonder what proportion of backed up files and systems are actually in a state to be restore…

> I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups. If you can afford 12 months of backups then you can almost certainly afford >=12 years of yearly backups by buying a new set of monthly tapes once a…

But you are then archiving data (not just backing it up), and will need to work out how to delete records you are no longer legally able to keep.

(GDPR and earlier laws, etc.)

Post reply on HN