Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Ever worked with a service that can never be restarted?

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 201 posts

Re: Ask HN: Ever worked with a service that can never be restarted?

#121

Post by [deleted] »

jlokier wrote: post #81
verytrivial wrote: post #70

My advice: 1. Suggest that the work to replace it is prioritized commensurate with the business impact caused whilst re-establishing service as if it went down right now 2. Remind them that it will go down at the worst possible time. 3. Ensure your name is attached to these two warnings. 4. Promise yourself you wouldn't run your business this way. 5. Get on with your life.

I would add to that: - Make a plan of exploratory steps that could trigger a failure, in order of increasing risks. - Then, if necessary, get a formal sign-off from someone with authority (perhaps a director of the company) to proceed with each step. Steps like that, which are usually very low risk but do risk taking down an unknown service, might include things like (just ideas from other comments): - Logging in to…

[deleted]

Re: Ask HN: Ever worked with a service that can never be restarted?

#122

Post by mst »

scandox wrote: post #42
arethuza wrote: post #27

Earlier quoted context omitted.

Have you tried decompiling the executable to see if any sensible information can be gained that way?

Everyone's afraid to do anything on that machine at all. So even getting agreement to login or install software or run tcpdump or whatever is fraught.

It might not help much but maybe you can get permission to run tcpdump on a machine that talks to the service?

Re: Ask HN: Ever worked with a service that can never be restarted?

#123

Post by aasasd »

> In addition the source code for this compiled binary has also been lost in the mists of time.

Sounds like you could try copying the binary, putting it into an isolated VM—or a virtual copy of your other services—and beating a file with a stick until the service accepts it as a config.

Re: Ask HN: Ever worked with a service that can never be restarted?

#124

Post by 3fe9a03ccd14ca5 »

verytrivial wrote: post #70

My advice: 1. Suggest that the work to replace it is prioritized commensurate with the business impact caused whilst re-establishing service as if it went down right now 2. Remind them that it will go down at the worst possible time. 3. Ensure your name is attached to these two warnings. 4. Promise yourself you wouldn't run your business this way. 5. Get on with your life.

Follow up advice:

After sufficient CYA steps are taken, don’t offer any help on this problem unless explicitly asked to do so. And even then wait until there’s actually desperation. Feel free to think about it so you have some ideas when the time comes, but if this service isn’t something you are responsible for then stay away.

Or else it will be a “you touched it last you broke it” regardless of what actually happened. And if you make too much ruckus about it, they’ll say “why didn’t you do anything to fix it??”

Re: Ask HN: Ever worked with a service that can never be restarted?

#126

Post by sdmike1 »

There is a tool used in malware analysis and computer forensics called Volatility[0]. It has some very powerful analysis tools and works on Linux Mac and Windows. In your case its ability to dump the memory of a running process without messing with the process state [1] may be very helpful! It also has the ability to run a Yara scan against the dumped memory which could let you find the region of memory containing the config file (so long as you know some of the strings in it).

Hope this helps!

[0] https://github.com/volatilityfoundation/volatility

[1] https://www.andreafortuna.org/2017/07/10/volatility-my-own-c...

Re: Ask HN: Ever worked with a service that can never be restarted?

#127

Post by ratel »

aequitas wrote: post #64
ratel wrote: post #62

Earlier quoted context omitted.

I think your solution is fine. Just another perspective on that server message. This server might be actually running a real-time legacy interface to one of your biggest customers who pay a 6 figure premium to keep this old thing running, just so they have time to migrate. Which they have been trying to do since the 80's, hence the fact that few people remember why it is there. I'm not saying this happened here, but…

> just so they have time to migrate. Which they have been trying to do since the 80's, hence the fact that few people remember why it is there. The questions I would ask are: what is the insurance if the system fails due to natural causes (hardware failure, power outage, you name it)? And what is the cost/benefit balance of having this unpatched security hole connected to the internet (which was the reason we discove…

To be sure I'm not defending the practice, just shedding another light on it.

In answer to your first question. There might not be an insurance against the system failing. The honest answer to what to do then, because someone always needs to ask is: Panic! It might not even be unrecoverable, just really expensive and time-consuming to do so.

The follow-up question would be: Yes it is a risk, yes it is a possible disaster. What do you want to do about it? If your answer is anything more than: We should not have taken on that risk in the first place or lets bet the companies future on the fact that we can fix this. Then it might actually be interesting to listen to. It seems from your comment you resolved it.

As for messaging. I do not agree you should explain the situation on the server. Just let people know they should not touch this ever. As soon as you explain why people will assume their reason to do so will trump whatever reason you gave. If they need to think of all possible disasters that might happen it has more impact than the one you can describe.

Messages like "Before doing anything on this server contact Bob" will eventually lead to Bob receiving a message: "We have done this or that. Just letting you know, but it was after office hours" which he will probably see rushing into the office in the middle of the night because the server is not working anymore. The other type of message: "Don't reboot this server ever! For more information contact Bob" Bob changed because he spend a considerable time saying no and explaining the situation to the sysops team, their manager, their manager's manager, etc. who all thought their priority must trump Bob's. Bob might still be working for the company. He might have tried the better part of his career to get this stain resolved. Nothing bad about Bob.

Re: Ask HN: Ever worked with a service that can never be restarted?

#128

Post by dboreham »

jlokier wrote: post #87

I have used GDB to look at a process and get a dump of particular run-time data structures, which is usually enough to reconstruct a config file. Config data structures usually don't change while a process is running. Often they are just values in global variables. If you have the executable file for the process, it may be possible to run that with trial config files, and then compare the GDB dump from the running se…

I was going to post roughly this, but with one addition: find a human with knowledge of the config file format. Doesn't have to be a perfect memory but surely someone developed this application and will remember roughly what went in the file and its layout. Then use "strings" or equivalent on the binary and your memory dump to piece together a reproduced config file. Test that in a separate VM until it works. Biggest problem is probably going to be the cut over between the running and new config file instances. If it doesn't work you're screwed. Perhaps do that by diverting network traffic, leaving the working legacy instance in place to allow fall back?

Re: Ask HN: Ever worked with a service that can never be restarted?

#129

Post by guiriduro »

Are you working on a Nuclear Reactor or something that will cause loss of life if rebooted accidentally? If yes, then you have a truly critical system that needs very careful uptime management, despite huge costs to carefully derisk and duplicate it, and there's plenty of good advice here already. But too many systems are 'super pets' like this and are mistakenly considered critical at exorbitant cost.

If no: turn it off and on, and see. The simple fact of the matter is that sooner or later it will happen anyway, and its better to bring that about, learn and solve. And if it results in large financial losses for extended downtime, then the management that allowed it to get to that state is already at fault (not you) and some better, safer alternative will arise from your efforts. Don't sweat it.

Re: Ask HN: Ever worked with a service that can never be restarted?

#130

Post by alexandercrohde »

ThomasRedstone wrote: post #40

A lot of talk about what to do yourself, I'd say do nothing yourself. Find an expert who has masses of experience who can consult on it. This isn't a good time to be learning and testing those lessons.

Yes. Moreover, if you backup the system great, you get peace of mind, but management hardly cares. If you BREAK the system while trying to establish a backup, management cares and it's "your fault."

You have exponentially more to lose than to gain by experimenting with this at all. In this situation you should convey the risks, let the stakeholders decide if/who is going to address this, as it's their risk to take.

Post reply on HN