You could try to make every possible request programmatically & save the responses & then rebuild it.
Ask HN: Ever worked with a service that can never be restarted?
11–20 of 201 posts
Re: Ask HN: Ever worked with a service that can never be restarted?
#12In case you want to replace or upgrade the UPS note they make devices that can cut into the power cord used by law enforcement to move a computer which is being impounded without depowering it.
Are you saying there is a device to splice the power cable while it is still hot, and re-route power to the computer without causing a hiccup?
I think they could run wires to the motherboard pins on the backside or something too
Re: Ask HN: Ever worked with a service that can never be restarted?
#13At a previous job I once sshd into a legacy system which had a motd in the vein of: "Don't ever reboot this system, if you do, I will find you and do horrible things to you". No explaination of why. My opinion was that the person who dared to leave a system in this state was the one who needed the horrible things done to them instead. Needless to say, he was already fired.
We started working on solution to run parallel to the existing one, which would receive shadow traffic so we could observe behaviour, find all edge cases and put them in a testsuite for the new solution. After we where confident that our testsuite contained most of the important behaviour we switched traffic to our new solution, keeping the old online just in case we needed to switch back.
Key is monitoring and learning the expected behaviour of the connected systems, to sense if that behaviour is not as expected and being able to act upon that as soon as possible.
Re: Ask HN: Ever worked with a service that can never be restarted?
#14Actually, I have experience here. The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is going to be met with heavy criticism. So you must be careful, the company has accepted the risk that they're in and you'll have to contend with that most likely. First thing is first: is it a VM or a Physical machine? Things get a little easier if it's a VM be…
> because the one thing that I learn as I learn more about computers is that they're basically working "by accident"
So true. Thanks this is good advice. As it happens it is a physical server.
I'm not actually mandated to do anything except replace it with a comparable service but this makes me feel like I'm racing against time!
Re: Ask HN: Ever worked with a service that can never be restarted?
#15I'm not sure that 55 months of up-time indicates it's more or less likely to go down in the next month, but I'd guess more likely. Surely there are options, have you tried de-compiling the source from the binary? >but the physical config was accidentally overwritten and there are no backups Any old dev PCs lying around somewhere? It's worth reaching out to the old developers to see if they have a copy, in really old…
> reaching out to the old developers to see if they have a copy
I think you mean "older devs that are still with the company" but I was once contacted by a former employer for exactly this: they had lost source code for the billing system. I didn't have the source code so that was easy to answer. Developers that have stolen intellectual property from former employers may want to consider carefully how they answer such questions...
Also, I was never supposed to have had access to this code whilst I was employed so I did wonder if there was more to it than just "oops we lost it can you help?". Perhaps they were trying to chase down a leak or something? I never heard back though (they survived).
Re: Ask HN: Ever worked with a service that can never be restarted?
#16Actually, I have experience here. The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is going to be met with heavy criticism. So you must be careful, the company has accepted the risk that they're in and you'll have to contend with that most likely. First thing is first: is it a VM or a Physical machine? Things get a little easier if it's a VM be…
I have a personal theory that Really Bad Things (TM) usually tend to happen during the response to other, relatively minor, problems.
So whatever is done cannot risk the running process. So personally I'd be loathed to try anything that touches the running process - even if they seem low risk.
Edit: One thing I would do is make it clear that building a replacement and keeping this thing running are different responsibilities and if you are doing the former then you really shouldn't be held responsible for the latter (politics I know).
Re: Ask HN: Ever worked with a service that can never be restarted?
#17May I ask about the business you are in? What happens to the world if your service goes down? It's just curiosity. Hopefully not related to breathing machines like importance. :)
Re: Ask HN: Ever worked with a service that can never be restarted?
#18Actually, I have experience here. The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is going to be met with heavy criticism. So you must be careful, the company has accepted the risk that they're in and you'll have to contend with that most likely. First thing is first: is it a VM or a Physical machine? Things get a little easier if it's a VM be…
> because the one thing that I learn as I learn more about computers is that they're basically working "by accident" So true. Thanks this is good advice. As it happens it is a physical server. I'm not actually mandated to do anything except replace it with a comparable service but this makes me feel like I'm racing against time!
[deleted]
Re: Ask HN: Ever worked with a service that can never be restarted?
#19Rather than focus on a replacement they should focus on reading the systems memory out and creating a replacement virtual machine or perhaps even trying to decipher the config structure from memory.
My experience with migration projects is that they can drag on in time, and all the while this system is just itching to go down due to a power failure or some other issue.
Re: Ask HN: Ever worked with a service that can never be restarted?
#20I'm not sure that 55 months of up-time indicates it's more or less likely to go down in the next month, but I'd guess more likely. Surely there are options, have you tried de-compiling the source from the binary? >but the physical config was accidentally overwritten and there are no backups Any old dev PCs lying around somewhere? It's worth reaching out to the old developers to see if they have a copy, in really old…
> I'm not sure that 55 months of up-time indicates it's more or less likely to go down in the next month, but I'd guess more likely.
If 'going down' is an event governed by chance (e.g. power failure), then it does not matter if it has been up for 55 months or 55 minutes.