Ask HN: Ever worked with a service that can never be restarted?

I would add to that:

- Make a plan of exploratory steps that could trigger a failure, in order of increasing risks.

- Then, if necessary, get a formal sign-off from someone with authority (perhaps a director of the company) to proceed with each step.

Steps like that, which are usually very low risk but do risk taking down an unknown service, might include things like (just ideas from other comments):

- Logging in to the server at all.

- Splicing something into the network switch.

- Cloning filesystems or disk images.

- Cloning the process memory or system memory.

- Cloning the image or other things if it's in a VM.

- Running strace gdb, or packet tracing.

- Swapping the power source live.

These days, it is often possible to transfer a full working system into a VM on a more modern, powerful machine, and after doing that, it's a great relief to everyone because it's no longer hardware dependent.

I've done that with some legacy systems that were originally on bare metal, and are now still running nearly 20 years later in reliable VMs, with no change to the running software. Usually running faster and with more memory doesn't break a working system.

But doing it on a service that can't be taken down even for a moment is quite an interesting adventure! :-) It is possible, but technically challenging, as long as you can obtain disk and memory images from the live system and then capture changes fast enough to perform a hot transfer.

> For servers I expect that they would have redundant inputs, that you can easily hot swap it.

This is true, but there is still a real risk that the dual power input on the server fails when you actually do the swap.

For example, the second PSU may not work, or it may have failed in the past and then be forgotten about, or connector or cable internally may be flaky, or the component that combines power from both PSUs may fail (although I think that's quite rare).

If it's a really critical service, those risks may be unacceptable.

You can use /proc to get what you need. It has a memory map and the memory itself. Read the map, pull the heap segments out, and read them into files named after the addresses. On an identical machine, load the process up, attach via gdb, and (this is the tricky part) then mmap in the files saved from the other side at the right addresses. If you can get the roots into your heap (tough), you can use gdb'd python API to write a traverser to get the data you need back out.

I have used GDB to look at a process and get a dump of particular run-time data structures, which is usually enough to reconstruct a config file.

Config data structures usually don't change while a process is running. Often they are just values in global variables.

If you have the executable file for the process, it may be possible to run that with trial config files, and then compare the GDB dump from the running service with the GDB dump from a trial config, to compare the relevant data structures. That can provide more confidence than just figuring out what the config ought to be.

Getting a GDB dump of the running service will be quite disruptive if it's done manually, but that might not matter. It will depend on whether the service is heavily used or if it's only doing something occasionally.

If the service is in constant use, it could make sense to automate GDB so it captures the config data structures quickly then safely detaches, and only briefly pauses the service.

Alternatively, if even automated GDB is too disruptive or difficult to use, or if the ptrace() syscalls used by GDB might cause a problem, it is often possible to capture a memory dump of the running process without affecting the process much, via /proc/PID/mem on Linux.

If necessary, write a tool in C to walk the heap by reading it from /proc/PID/mem to reconstruct the config data structures that way.

(All the above assumes Linux.)

From my experience, dumping memory and most likely reverse engineering the binary is a must.

Actually as it is legacy system, changes are this will be easier.

To be honest, the path to follow mostly depends on the OS.

As it is critical system, I would start with reverse engineering the binary, making sure the config is preserved across life time of application.