Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Ever worked with a service that can never be restarted?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 201 posts

Re: Ask HN: Ever worked with a service that can never be restarted?

#51

Post by jerome-jh »

goatinaboat wrote: post #35
exdsq wrote: post #33

Earlier quoted context omitted.

Is this seriously done? I just completed a small literature review for neural program synthesis which uses inputs as training data to generate a program using neural networks that can generate the right outputs but it's a huge search problem and I assume for anything non-trivial this is out of our reach? As an example, Deepmind achieved binary multiplication using this technique in 2018 and this was notable.

I’ve never done it but it should be feasible. An IP packet is 1518x8 == 12144 bits which is not especially big for a one-hot style encoding these days. You would just need to be sure to capture representative traffic e.g. accounting for monthly jobs.

In my understanding, ML works by mapping a huge input space to a tiny output space, and most of the input space remains unexplored.

In this case the input space is small and you want a quite dense mapping from input to output. I do not think ML applies.

Re: Ask HN: Ever worked with a service that can never be restarted?

#52

Post by jerome-jh »

goatinaboat wrote: post #29

capture packet's and see what kind of traffic it receives most commonly The modern solution would be to capture the incoming packets as a training set then apply machine learning to create a model that can perfectly recreate the outgoing packets. It’s still an inexplicable black box of course, that is the nature of ML, but at least you can run it in the cloud now.

You end up in the same state you started from: running a blackbox.

Re: Ask HN: Ever worked with a service that can never be restarted?

#53

Post by dijit »

pjc50 wrote: post #21
dijit wrote: post #7

Actually, I have experience here. The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is going to be met with heavy criticism. So you must be careful, the company has accepted the risk that they're in and you'll have to contend with that most likely. First thing is first: is it a VM or a Physical machine? Things get a little easier if it's a VM be…

Attaching a debugger potentially disrupts the service. I would suggest trying to live-image it with some other technique first.

Yeah, that's pretty much the last resort once you have fallbacks in place, might be the last/only way to glean info about the config that it's running.

Re: Ask HN: Ever worked with a service that can never be restarted?

#54

Post by goatinaboat »

jerome-jh wrote: post #52
goatinaboat wrote: post #29

capture packet's and see what kind of traffic it receives most commonly The modern solution would be to capture the incoming packets as a training set then apply machine learning to create a model that can perfectly recreate the outgoing packets. It’s still an inexplicable black box of course, that is the nature of ML, but at least you can run it in the cloud now.

You end up in the same state you started from: running a blackbox.

True but now you are independent of the original hardware.

Re: Ask HN: Ever worked with a service that can never be restarted?

#55

Post by lormayna »

At my previous-previous job we had a core switch that was on from 5 years. No one wants to reload because everyone was scared it cannot restart anymore. We don't have maintenance, spare parts and cabling was a mess.

My colleagues told me that it was still there 2 years ago.

Re: Ask HN: Ever worked with a service that can never be restarted?

#56

Post by [deleted] »

goatinaboat wrote: post #29

capture packet's and see what kind of traffic it receives most commonly The modern solution would be to capture the incoming packets as a training set then apply machine learning to create a model that can perfectly recreate the outgoing packets. It’s still an inexplicable black box of course, that is the nature of ML, but at least you can run it in the cloud now.

[deleted]

Re: Ask HN: Ever worked with a service that can never be restarted?

#57

Post by quickthrower2 »

goatinaboat wrote: post #54
jerome-jh wrote: post #52

Earlier quoted context omitted.

You end up in the same state you started from: running a blackbox.

True but now you are independent of the original hardware.

Ha ha. If it’s that easy why developer software at all. Just run a NN on packers and you can create anything without coding!

Re: Ask HN: Ever worked with a service that can never be restarted?

#58

Post by bouke »

dijit wrote: post #8
blackrock wrote: post #6

Earlier quoted context omitted.

Are you saying there is a device to splice the power cable while it is still hot, and re-route power to the computer without causing a hiccup?

Oh for sure this is possible: https://www.youtube.com/watch?v=vQ5MA685ApE the devices that do it without doing what these guys did is quite pricey though https://www.cru-inc.com/products/wiebetech/hotplug_field_kit...

That requires the computer to hotswap to be plugged into a power strip. It doesn't work if the computer is plugged into the wall directly. (Well leaving aside modifying the house's electricity / opening up the wall obviously.)

Re: Ask HN: Ever worked with a service that can never be restarted?

#59

Post by ratel »

I have some experience in this field.

First of all you are there to build a replacement service. As long as you do that in the allotted time what happens to the old should not bother you. Focus on your part of the solution.

The fact that you do seem to worry suggests you need something from the old service. Do you have a good design for the new service? Are there details of the old service that you do not know about? It won't be the first time a company has an application that runs complex calculations, but nobody knows how it is done. We actually built black-box applications in the past to replace calculation modules on obsolete hardware.

If you need information from the old system you may have to resort to cloning the input and output preferably on the network level so you don't have to mess with the existing service.

If the replacement service is years out you might suggest to the company to put in a red team. Another person or persons who work on finding out how the service works by running a duplicate on another system, rent it if necessary, and poking at it seeing if they can get comparable results quicker than the replacement service. But that is not you.

There are a couple of things you need to have in place before going live. Make sure you can clone input and output so you can run the old service next to the new one for a while. You don't want to stop the old service the moment you go live, because if anything goes wrong you need to be able to switch back. Even than starving the old service of input might have adverse effects, so cloning is better. If the old service uses external resources, like a database, files, etc. make sure you do not interfere with file locks, sequence numbers,etc. when running in parallel.

Re: Ask HN: Ever worked with a service that can never be restarted?

#60

Post by exdsq »

quickthrower2 wrote: post #57
goatinaboat wrote: post #54

Earlier quoted context omitted.

True but now you are independent of the original hardware.

Ha ha. If it’s that easy why developer software at all. Just run a NN on packers and you can create anything without coding!

This problem is being worked on but it is by no means solved!

This paper [0] is a pretty decent survey on the field and [1] is an excellent article on the topic at a high level, written by a team currently working in the space (synthesising smart contracts).

[0] https://arxiv.org/pdf/1802.02353.pdf

[1] https://medium.com/@vidiborskiy/software-writes-software-pro...

Post reply on HN