Live data from Hacker News

Ask HN: Ever worked with a service that can never be restarted?

news.ycombinator.com

1–10 of 201 posts

Ask HN: Ever worked with a service that can never be restarted?

#1
I'm currently working with a legacy system. One element of it has a loaded config in memory, but the physical config was accidentally overwritten and there are no backups. In addition the source code for this compiled binary has also been lost in the mists of time.

The service has current up time of 55 Months. The general consensus therefore is that as long as it is never restarted it will continue to perform its function until a replacement can be put in place. Which seems a little fatalistic to me...

Has anyone experience of doing something sensible in a similar situation?

Re: Ask HN: Ever worked with a service that can never be restarted?

#2
Dump memory and see if you can retrieve the config that's running. If someone has a clue about how the config should be, you could probably get all the elements of it from simply running 'strings' on the binary. You could also disassemble the binary. Possibly the deleted config file could be salvaged from disk.

As is, this is all relying on the HW not giving up and the UPS working in a power failure!

Re: Ask HN: Ever worked with a service that can never be restarted?

#6

In case you want to replace or upgrade the UPS note they make devices that can cut into the power cord used by law enforcement to move a computer which is being impounded without depowering it.

Are you saying there is a device to splice the power cable while it is still hot, and re-route power to the computer without causing a hiccup?

Re: Ask HN: Ever worked with a service that can never be restarted?

#7
Actually, I have experience here.

The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is going to be met with heavy criticism. So you must be careful, the company has accepted the risk that they're in and you'll have to contend with that most likely.

First thing is first: is it a VM or a Physical machine? Things get a little easier if it's a VM because there might be vMotion or some kind of live-migration in place meaning hardware failure might not actually bring the service down.

Next thing is you absolutely have to plan for failure because the one thing that I learn as I learn more about computers is that they're basically working "by accident", so much error correction goes into systems that it absolutely boggles my mind and failures still get through. So, it's certainly not a question of "if" but "when" and plan for it being soon.

Now, the obvious technical things are:

* Dump the memory * Grab any files it has open from its file-descriptors (/proc//fd/) Your config file might be there... but somehow I doubt it. Attach a debugger and dump as much state as possible

Be sure to cleanly detatch : https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_22.htm...

Don't use breakpoints! They will obviously halt execution.

If it was my service I would also capture packet's and see what kind of traffic it receives most commonly; and I would make some kind of stub component to replace it before working on the machine. Just in case I break it and it causes everything to go down that depends on it.

But, this is a horrible situation, reverse engineering this thing is going to be a pain. Good Luck.

Re: Ask HN: Ever worked with a service that can never be restarted?

#8

In case you want to replace or upgrade the UPS note they make devices that can cut into the power cord used by law enforcement to move a computer which is being impounded without depowering it.

Are you saying there is a device to splice the power cable while it is still hot, and re-route power to the computer without causing a hiccup?

Oh for sure this is possible: https://www.youtube.com/watch?v=vQ5MA685ApE

the devices that do it without doing what these guys did is quite pricey though https://www.cru-inc.com/products/wiebetech/hotplug_field_kit...

Re: Ask HN: Ever worked with a service that can never be restarted?

#9
I'm not sure that 55 months of up-time indicates it's more or less likely to go down in the next month, but I'd guess more likely.

Surely there are options, have you tried de-compiling the source from the binary?

>but the physical config was accidentally overwritten and there are no backups

Any old dev PCs lying around somewhere? It's worth reaching out to the old developers to see if they have a copy, in really old legacy systems it's fairly likely someone will have a copy somewhere.

Re: Ask HN: Ever worked with a service that can never be restarted?

#10
post #8

Earlier quoted context omitted.

Are you saying there is a device to splice the power cable while it is still hot, and re-route power to the computer without causing a hiccup?

Oh for sure this is possible: https://www.youtube.com/watch?v=vQ5MA685ApE the devices that do it without doing what these guys did is quite pricey though https://www.cru-inc.com/products/wiebetech/hotplug_field_kit...

I was thinking more along the lines of a single power input like your home desktop PC. That you would need to splice into this power cable while it’s still hot.

For servers I expect that they would have redundant inputs, that you can easily hot swap it.

The second link had a video on how to do that. With multiple ways to achieve a power re-route, including a tool to easily splice into the wire. The easiest technique was to just use a sheath to touch the metal prongs on the cable.

But the surprise for me was the power strip technique, where he plugged in his battery source into one of the 6 outlets, thereby simulating a power input. I didn’t even know that was possible.

Post reply on HN