Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Ever worked with a service that can never be restarted?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 201 posts

Re: Ask HN: Ever worked with a service that can never be restarted?

#61

Post by dijit »

bouke wrote: post #58
dijit wrote: post #8

Earlier quoted context omitted.

Oh for sure this is possible: https://www.youtube.com/watch?v=vQ5MA685ApE the devices that do it without doing what these guys did is quite pricey though https://www.cru-inc.com/products/wiebetech/hotplug_field_kit...

That requires the computer to hotswap to be plugged into a power strip. It doesn't work if the computer is plugged into the wall directly. (Well leaving aside modifying the house's electricity / opening up the wall obviously.)

you can do it pretty easily if you have a dual socket wall socket because it follows the exact same principles. But you have to be very careful.

In these cases it's probably worth doing it though.

https://media.screwfix.com/is/image//ae235?src=ae235/38040_P...

FWIW I do not endorse doing this, just stating that its possible and would work, it's not electrically safe and is dangerous for not only the hardware but for you too.

Re: Ask HN: Ever worked with a service that can never be restarted?

#62

Post by ratel »

aequitas wrote: post #13

At a previous job I once sshd into a legacy system which had a motd in the vein of: "Don't ever reboot this system, if you do, I will find you and do horrible things to you". No explaination of why. My opinion was that the person who dared to leave a system in this state was the one who needed the horrible things done to them instead. Needless to say, he was already fired. We started working on solution to run parall…

I think your solution is fine.

Just another perspective on that server message. This server might be actually running a real-time legacy interface to one of your biggest customers who pay a 6 figure premium to keep this old thing running, just so they have time to migrate. Which they have been trying to do since the 80's, hence the fact that few people remember why it is there.

I'm not saying this happened here, but it happens a lot more than any of us want. Just keep in mind there might actually be a good reason for the warning, although I would object to the wording. First head the warning. Then figure out why it is there.

Re: Ask HN: Ever worked with a service that can never be restarted?

#64

Post by aequitas »

ratel wrote: post #62
aequitas wrote: post #13

At a previous job I once sshd into a legacy system which had a motd in the vein of: "Don't ever reboot this system, if you do, I will find you and do horrible things to you". No explaination of why. My opinion was that the person who dared to leave a system in this state was the one who needed the horrible things done to them instead. Needless to say, he was already fired. We started working on solution to run parall…

I think your solution is fine. Just another perspective on that server message. This server might be actually running a real-time legacy interface to one of your biggest customers who pay a 6 figure premium to keep this old thing running, just so they have time to migrate. Which they have been trying to do since the 80's, hence the fact that few people remember why it is there. I'm not saying this happened here, but…

> just so they have time to migrate. Which they have been trying to do since the 80's, hence the fact that few people remember why it is there.

The questions I would ask are: what is the insurance if the system fails due to natural causes (hardware failure, power outage, you name it)? And what is the cost/benefit balance of having this unpatched security hole connected to the internet (which was the reason we discovered this).

If a system is so important that such a warning is deemed appropriate, imho its just laziness and carelessness from the warning's writer and they are shifting their problem onto the next person that stumbles upon the system.

At least give some information on why it is important and at what time or under which conditions the warning can be considered expired. Just like commenting your code is important because you never know who is having to make sense of it in a few months time (especially true if that person turn out to be you :)).

Re: Ask HN: Ever worked with a service that can never be restarted?

#65

Post by lmilcin »

dijit wrote: post #7

Actually, I have experience here. The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is going to be met with heavy criticism. So you must be careful, the company has accepted the risk that they're in and you'll have to contend with that most likely. First thing is first: is it a VM or a Physical machine? Things get a little easier if it's a VM be…

I would add: overcommunicate everything you plan before you do and give others plenty of time to react to even simplest operations.

The company may have failed but when the proverbial truckload of shit hits the proverbial supersonic fan there will be tremendous pressure to find a culprit. A small peon might find himself/herself outnumbered and outgunned.

Re: Ask HN: Ever worked with a service that can never be restarted?

#66

Post by pjc50 »

ratel wrote: post #62
aequitas wrote: post #13

At a previous job I once sshd into a legacy system which had a motd in the vein of: "Don't ever reboot this system, if you do, I will find you and do horrible things to you". No explaination of why. My opinion was that the person who dared to leave a system in this state was the one who needed the horrible things done to them instead. Needless to say, he was already fired. We started working on solution to run parall…

I think your solution is fine. Just another perspective on that server message. This server might be actually running a real-time legacy interface to one of your biggest customers who pay a 6 figure premium to keep this old thing running, just so they have time to migrate. Which they have been trying to do since the 80's, hence the fact that few people remember why it is there. I'm not saying this happened here, but…

If that were true, they should have put it in the motd as well.

Re: Ask HN: Ever worked with a service that can never be restarted?

#67

Post by lexicality »

dijit wrote: post #8
blackrock wrote: post #6

Earlier quoted context omitted.

Are you saying there is a device to splice the power cable while it is still hot, and re-route power to the computer without causing a hiccup?

Oh for sure this is possible: https://www.youtube.com/watch?v=vQ5MA685ApE the devices that do it without doing what these guys did is quite pricey though https://www.cru-inc.com/products/wiebetech/hotplug_field_kit...

In the grand scheme of things, $600 is nothing money to a company.

Re: Ask HN: Ever worked with a service that can never be restarted?

#68

Post by crusso »

exdsq wrote: post #33
goatinaboat wrote: post #29

capture packet's and see what kind of traffic it receives most commonly The modern solution would be to capture the incoming packets as a training set then apply machine learning to create a model that can perfectly recreate the outgoing packets. It’s still an inexplicable black box of course, that is the nature of ML, but at least you can run it in the cloud now.

Is this seriously done? I just completed a small literature review for neural program synthesis which uses inputs as training data to generate a program using neural networks that can generate the right outputs but it's a huge search problem and I assume for anything non-trivial this is out of our reach? As an example, Deepmind achieved binary multiplication using this technique in 2018 and this was notable.

Is this seriously done?

No. Not even close for any general purpose I/O.

Re: Ask HN: Ever worked with a service that can never be restarted?

#69

Post by martin_a »

goatinaboat wrote: post #29

capture packet's and see what kind of traffic it receives most commonly The modern solution would be to capture the incoming packets as a training set then apply machine learning to create a model that can perfectly recreate the outgoing packets. It’s still an inexplicable black box of course, that is the nature of ML, but at least you can run it in the cloud now.

Sorry that nobody else got it, but I like that joke.

Do you think we can put some crypto in there, too?

Re: Ask HN: Ever worked with a service that can never be restarted?

#70

Post by verytrivial »

My advice:

1. Suggest that the work to replace it is prioritized commensurate with the business impact caused whilst re-establishing service as if it went down right now

2. Remind them that it will go down at the worst possible time.

3. Ensure your name is attached to these two warnings.

4. Promise yourself you wouldn't run your business this way.

5. Get on with your life.

Post reply on HN