Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Ever worked with a service that can never be restarted?

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 201 posts

Re: Ask HN: Ever worked with a service that can never be restarted?

#111

Post by setr »

nnq wrote: post #92

Unpopular advice: find a subtle way to make it crash, preferably stealthy, but if not possible, at least in a way that can be attributable to mild innocent incompetence instead of malice! Then there will be more and more interesting work to do for you and others, either rediscovering and properly documenting the config, or, hopefully, architecting and coding its replacement! In the aftermath, the organization will be…

This is naive evil Mastermind level thinking, where the villain spouts his whole line and you think "no wait that's just stupid.." as he sends off the sequence of nukes

>If it actually collapses bc of this, then it deserved to die anyway, you only helped accelerate the outcome and reduced the suffering

If you want to make an evolutionary "survival of the fittest" type of claim... Life is entirely about error correction (and reproduction + staving off death as long as possible). Engineers babbling to engineers is part of a business's error correction.. Duct tape and monkeypatching is part of the engineer's error correction..

A rogue engineer trying to invoke rapid deconstruction of his environment (a virus) is one of those errors -- Solution: engage, restrain, and eject.

The legitimate error correction is already happening, by means of hope & replace. Presumably because other business concerns make it so that actual shutdown would be a Terrible, Horrible, No Good, Very Bad idea.

>If it actually collapses bc of this, then it deserved to die anyway, you only helped accelerate the outcome and reduced the suffering.

A doctor can't help you if you die in minutes. They can save you if you die in hours. Time is a major part of error correction.

Oh damn, you've pushed out the nukes already.

>Some things and processes need to be "helped to fail faster", everyone will benefit from the renewal in the end, even if most will hate it ;)

Seems to me the main process requiring failure acceleration is this one employee's UAC expiration policy.

Re: Ask HN: Ever worked with a service that can never be restarted?

#112

Post by martin_a »

hoorayimhelping wrote: post #95
martin_a wrote: post #69

Earlier quoted context omitted.

Sorry that nobody else got it, but I like that joke. Do you think we can put some crypto in there, too?

> Sorry that nobody else got it, but I like that joke. No, plenty of us got it. It's just not funny right now. A person is trying to find a real solution to a real problem they have. Every comment in this thread represents a potential hope that someone knows how to fix it. When someone comes in to a serious thread that is spitballing ideas and tries to make it about them by cracking jokes and being fun and clever, it…

> It's noise.

Just like your reply.

Besides that: OP is fucked and left with a huge pile of technical debt. While his intention to help and find a solution is noble, one should face the truth: One way or another this system will go down and nobody will be able to restart it. Just leave it where it is and start building a replacement that is well documented and has backups. Any time spent on saving this system is thrown away. So: This is totally the time to make fun. Call it gallows humor or whatever but stop whining.

Re: Ask HN: Ever worked with a service that can never be restarted?

#113

Post by [deleted] »

celticmusic wrote: post #107
_jal wrote: post #98

Earlier quoted context omitted.

Back when I ran a company, my partners and I gamed out what to do with employees in various situations, as a way of coming to a consensus. I was against legal action in all but extreme hypotheticals. Behavior like this is in that class. You assume you not only have all the relevant information to predict the outcome, but also that your analysis of the situation is superior enough to trump those who actually own the p…

If you had hired competent people to begin with, the configs would have been backed up. I don't agree with the other posters recommendation, but I understand the sentiment. The same incompetence that allowed the event to happen is the same incompetence that allowed it to be swept under the rug for almost 6 years. I've seen paragons of hiring get upset because failures were made visible, not understanding that the fai…

[deleted]

Re: Ask HN: Ever worked with a service that can never be restarted?

#114

Post by nerpderp82 »

scandox wrote: post #42
arethuza wrote: post #27

Earlier quoted context omitted.

Have you tried decompiling the executable to see if any sensible information can be gained that way?

Everyone's afraid to do anything on that machine at all. So even getting agreement to login or install software or run tcpdump or whatever is fraught.

You need to setup an identical-ish host, could even be in VM so that you can build software that will run on the Ghost Ship. This in and of itself could be difficult, Debian 4?

Using this Shadow Ghost Ship you can build a copy of tcpdump that you can just scp over and run in place. Don't touch the package manager, /etc, /var, anything!

Re: Ask HN: Ever worked with a service that can never be restarted?

#115

Post by Piskvorrr »

"as long as it is never restarted"

Seen that. And then one day, power failed. UPSes kicked in. In one hour, still no mains power. Batteries depleted, systems started shutting down...including that one.

Had to build a replacement in a real hurry after that.

Re: Ask HN: Ever worked with a service that can never be restarted?

#116

Post by setr »

celticmusic wrote: post #107
_jal wrote: post #98

Earlier quoted context omitted.

Back when I ran a company, my partners and I gamed out what to do with employees in various situations, as a way of coming to a consensus. I was against legal action in all but extreme hypotheticals. Behavior like this is in that class. You assume you not only have all the relevant information to predict the outcome, but also that your analysis of the situation is superior enough to trump those who actually own the p…

If you had hired competent people to begin with, the configs would have been backed up. I don't agree with the other posters recommendation, but I understand the sentiment. The same incompetence that allowed the event to happen is the same incompetence that allowed it to be swept under the rug for almost 6 years. I've seen paragons of hiring get upset because failures were made visible, not understanding that the fai…

>If you had hired competent people to begin with, the configs would have been backed up

What happy-go-lucky universe do you hail from? You could hire the most competent people in the world and you'll still end up with a horribly complex shitshow with lost programs, untested backups, missing docs, implicit information out the ass, and so on. Software development has never had the discipline to claim compotent employees produce competent results -- they're just less likely to be shitty results.

And we spend 90÷ of our time dealing with every layer of incompetence our competent, and incompetent, friends have provided us

Re: Ask HN: Ever worked with a service that can never be restarted?

#117

Post by ZenPHP »

blackrock wrote: post #10
dijit wrote: post #8

Earlier quoted context omitted.

Oh for sure this is possible: https://www.youtube.com/watch?v=vQ5MA685ApE the devices that do it without doing what these guys did is quite pricey though https://www.cru-inc.com/products/wiebetech/hotplug_field_kit...

I was thinking more along the lines of a single power input like your home desktop PC. That you would need to splice into this power cable while it’s still hot. For servers I expect that they would have redundant inputs, that you can easily hot swap it. The second link had a video on how to do that. With multiple ways to achieve a power re-route, including a tool to easily splice into the wire. The easiest technique…

Hot swapping servers with redundant power supplies is great, until you discover the hard way that some nimrod plugged both power supplies into the same UPS. And not just once, but on every server in both racks, one UPS per server.

Re: Ask HN: Ever worked with a service that can never be restarted?

#118

Post by murph-almighty »

hoorayimhelping wrote: post #95
martin_a wrote: post #69

Earlier quoted context omitted.

Sorry that nobody else got it, but I like that joke. Do you think we can put some crypto in there, too?

> Sorry that nobody else got it, but I like that joke. No, plenty of us got it. It's just not funny right now. A person is trying to find a real solution to a real problem they have. Every comment in this thread represents a potential hope that someone knows how to fix it. When someone comes in to a serious thread that is spitballing ideas and tries to make it about them by cracking jokes and being fun and clever, it…

That just made me realize that my personal habit of joking during prod incidents, while relieving to me, might not be to co-workers. Thanks for the perspective check.

Re: Ask HN: Ever worked with a service that can never be restarted?

#119

Post by celticmusic »

setr wrote: post #116
celticmusic wrote: post #107

Earlier quoted context omitted.

If you had hired competent people to begin with, the configs would have been backed up. I don't agree with the other posters recommendation, but I understand the sentiment. The same incompetence that allowed the event to happen is the same incompetence that allowed it to be swept under the rug for almost 6 years. I've seen paragons of hiring get upset because failures were made visible, not understanding that the fai…

>If you had hired competent people to begin with, the configs would have been backed up What happy-go-lucky universe do you hail from? You could hire the most competent people in the world and you'll still end up with a horribly complex shitshow with lost programs, untested backups, missing docs, implicit information out the ass, and so on. Software development has never had the discipline to claim compotent employee…

It's possible to avoid losing a configuration.

Re: Ask HN: Ever worked with a service that can never be restarted?

#120

Post by [deleted] »

verytrivial wrote: post #70

My advice: 1. Suggest that the work to replace it is prioritized commensurate with the business impact caused whilst re-establishing service as if it went down right now 2. Remind them that it will go down at the worst possible time. 3. Ensure your name is attached to these two warnings. 4. Promise yourself you wouldn't run your business this way. 5. Get on with your life.

[deleted]
Post reply on HN