Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Ever worked with a service that can never be restarted?

news.ycombinator.com

Post reply on HN
Print view View original

161–170 of 201 posts

Re: Ask HN: Ever worked with a service that can never be restarted?

#161

Post by Nextgrid »

verytrivial wrote: post #70

My advice: 1. Suggest that the work to replace it is prioritized commensurate with the business impact caused whilst re-establishing service as if it went down right now 2. Remind them that it will go down at the worst possible time. 3. Ensure your name is attached to these two warnings. 4. Promise yourself you wouldn't run your business this way. 5. Get on with your life.

I would say keep your own copy of the warnings - email them to yourself or similar.

When shit hits the fan if the company wanted to throw you under the bus they could delete them remotely from your machine or work mailbox and then pretend the warnings never existed.

Re: Ask HN: Ever worked with a service that can never be restarted?

#162

Post by Nextgrid »

adrianmonk wrote: post #135
dijit wrote: post #7

Actually, I have experience here. The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is going to be met with heavy criticism. So you must be careful, the company has accepted the risk that they're in and you'll have to contend with that most likely. First thing is first: is it a VM or a Physical machine? Things get a little easier if it's a VM be…

I would add this: whatever you decide to do to the running binary, create a mock binary on another computer and try the exact sequence of commands/steps first. And verify that it doesn't disturb the mock. This should reduce risk a bit because you are not experimenting and creating your procedure on the fly when you do the actual thing. Instead, you are applying a procedure that you have rehearsed, and you have gained…

I would also recommend trying the process on a wide range of existing software you can get your hands on (let's say PostgreSQL or something similarly big). This will reduce the risk of any edge-case that your mock didn't cover.

Re: Ask HN: Ever worked with a service that can never be restarted?

#163

Post by marcosdumay »

DuskStar wrote: post #150
andai wrote: post #146

Earlier quoted context omitted.

What's special about mainframes in this context?

Mainframes tend to be designed with extreme uptime in mind - to the point of often having hot-swappable processors .

The software doesn't care. It will crash anyway.

Re: Ask HN: Ever worked with a service that can never be restarted?

#164

Post by marcosdumay »

surfsvammel wrote: post #157

Damn. Everyone here seems to advise you to just protect your own ass as a top priority. Is that a product of US work culture? If you where in a sound organisation, which I would say most Swedish IT organisations are, you need to think about the company and the clients first. Someone advised you to: “don’t do anything until explicitly asked to”. I think that’s just bad advise. You obviously know this is a major proble…

> If you where in a sound organisation

Most people here are arguing that if he was in a sound organization, that wouldn't happen. Thus, he isn't, and advice that assumes sound organizations is useless.

Re: Ask HN: Ever worked with a service that can never be restarted?

#165

Post by Ghjklov »

surfsvammel wrote: post #157

Damn. Everyone here seems to advise you to just protect your own ass as a top priority. Is that a product of US work culture? If you where in a sound organisation, which I would say most Swedish IT organisations are, you need to think about the company and the clients first. Someone advised you to: “don’t do anything until explicitly asked to”. I think that’s just bad advise. You obviously know this is a major proble…

Uh no, that's terrible advice. What if he ends up disrupting the service and cause days or worst weeks of downtime for the company? He'll end up taking the entire blame for it and possibly even fired. Will you come to back him up? Because to everyone else there, everything was working fine for months and years until OP came along and put his nose where he shouldn't have.

Re: Ask HN: Ever worked with a service that can never be restarted?

#168

Post by LifeLiverTransp »

Only tangential related, but do not delete files. Sort them into folders, compress those folders, archive those folders, but do not delete files. Everytime i cleaned up my machine, some customer jumped around the corner, asking for that specific project that i just purged. Yes, i have backups, but those backups are a) stowed away. b) maybee incomplete or out of date c) have been deleted by somebody else, who was tasked with a cleanup.

Re: Ask HN: Ever worked with a service that can never be restarted?

#169

Post by viraptor »

tylermac1 wrote: post #99
viraptor wrote: post #39

Earlier quoted context omitted.

This is not a reasonable approach at this point. You'd need to know the whole state required for processing. That includes precise time, any local storage, any remote dependencies, randomness seeds, etc. Even then, we're not even close to full synthesis from examples. This is not a modern solution. This is made up, beyond toy examples.

I think they're joking.

The follow-up sounded pretty serious: https://news.ycombinator.com/item?id=22063160

Re: Ask HN: Ever worked with a service that can never be restarted?

#170

Post by celticmusic »

setr wrote: post #159
celticmusic wrote: post #141

Earlier quoted context omitted.

> rather you need continuous, unfailing competency That's called doing it manually. What you're doing is making excuses rather than holding yourself to a higher level of quality.

>That's called doing it manually. Yes, a common feature in unstable environments. Automation is an ideal, but not a necessity, and it's not always viable or cost-effective. But also that the unfailing competency means that you need to contiously maintain (all aspects of the) automation without fail, again in spite of a changing environment. Across everyone who does things. >What you're doing is making excuses rather…

> Automation is an ideal, but not a necessity

You're just bad. You automate specifically to take the human error risk out of the equation, that's a part of what it means to be competent. We don't automate our deploys because it's an ideal that is not a necessity, but so some jackass doesn't delete a configuration while also never putting it into a backup, or source control, or on their dev machine, ad nauseum.

It's not an accident that some of us have been working 20+ years in this industry and have never been the cause of something like this.

> Automation is not an answer that can always be applied trivially; especially retroactively. Even despite all the marketing that will tell you otherwise.

shit, I guess I didn't read the right pamphlet...

Post reply on HN