Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Ever worked with a service that can never be restarted?

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 201 posts

Re: Ask HN: Ever worked with a service that can never be restarted?

#131

Post by dwnvoted2hell »

guiriduro wrote: post #129

Are you working on a Nuclear Reactor or something that will cause loss of life if rebooted accidentally? If yes, then you have a truly critical system that needs very careful uptime management, despite huge costs to carefully derisk and duplicate it, and there's plenty of good advice here already. But too many systems are 'super pets' like this and are mistakenly considered critical at exorbitant cost. If no: turn it…

Honestly, do this with a backup. Don't interrupt functionality and lose your job without a very good reason.

You should have a staging environment. Toss a clone in there and have someone get to work in reverse engineering.

Re: Ask HN: Ever worked with a service that can never be restarted?

#132

Post by gdxhyrd »

jerome-jh wrote: post #51
goatinaboat wrote: post #35

Earlier quoted context omitted.

I’ve never done it but it should be feasible. An IP packet is 1518x8 == 12144 bits which is not especially big for a one-hot style encoding these days. You would just need to be sure to capture representative traffic e.g. accounting for monthly jobs.

In my understanding, ML works by mapping a huge input space to a tiny output space, and most of the input space remains unexplored. In this case the input space is small and you want a quite dense mapping from input to output. I do not think ML applies.

No, ML works fine with large output spaces too.

The problem is that the solution space grows super exponentially, and if you need to find an exact one, then the number of samples too.

Re: Ask HN: Ever worked with a service that can never be restarted?

#133

Post by miohtama »

My father worked in a district heating plant. The control PC was 80286 and had uptime over 10 years. Though the plant in theory could be run without it with knobs and levers.

It was replaced by a Pentium with a virtualized MS-DOS due to y2k event, though I am quite sure nothing changed in the underlying MS-DOS program that was more or less time independent.

Re: Ask HN: Ever worked with a service that can never be restarted?

#134

Post by setr »

celticmusic wrote: post #119
setr wrote: post #116

Earlier quoted context omitted.

>If you had hired competent people to begin with, the configs would have been backed up What happy-go-lucky universe do you hail from? You could hire the most competent people in the world and you'll still end up with a horribly complex shitshow with lost programs, untested backups, missing docs, implicit information out the ass, and so on. Software development has never had the discipline to claim compotent employee…

It's possible to avoid losing a configuration.

It's possible, and it's possible to avoid losing source code, and its.possible to have valid backups, and keep your servers decemtly secure... yet it's sufficiently common (especially as the program ages) that its not trivially attributed to competency; rather you need continuous, unfailing competency, in the face of continuous environmental change (local changes, within the business, and external changes, as business needs change, and mergers occur, and people quit/fired, etc)

Re: Ask HN: Ever worked with a service that can never be restarted?

#135

Post by adrianmonk »

dijit wrote: post #7

Actually, I have experience here. The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is going to be met with heavy criticism. So you must be careful, the company has accepted the risk that they're in and you'll have to contend with that most likely. First thing is first: is it a VM or a Physical machine? Things get a little easier if it's a VM be…

I would add this: whatever you decide to do to the running binary, create a mock binary on another computer and try the exact sequence of commands/steps first. And verify that it doesn't disturb the mock.

This should reduce risk a bit because you are not experimenting and creating your procedure on the fly when you do the actual thing. Instead, you are applying a procedure that you have rehearsed, and you have gained partial confidence that it won't disturb a similar running binary.

The mock doesn't have to be a perfect reproduction of the real binary, but it should do similar things where relevant. For example, if the service listens on a TCP port, so should the mock. You might not have to code the mock from scratch; maybe some existing software is similar enough. Or maybe you can even run a non-production copy of your real binary, if you are really confident you can configure it to not interact with anything it shouldn't.

Re: Ask HN: Ever worked with a service that can never be restarted?

#137

Post by john_moscow »

sdmike1 wrote: post #126

There is a tool used in malware analysis and computer forensics called Volatility[0]. It has some very powerful analysis tools and works on Linux Mac and Windows. In your case its ability to dump the memory of a running process without messing with the process state [1] may be very helpful! It also has the ability to run a Yara scan against the dumped memory which could let you find the region of memory containing th…

I would advise exercising caution though. Even if the tool works perfectly fine, the service going down anywhere within days to weeks after you captured its state would be seen by the management as your fault. They won't care for the explanations, they will just need a scapegoat and that would be whoever touched the thing last time.

That said, there are plenty of legitimate (albeit low-probability) reasons for the dumping tool to screw up the service. Slightly affecting memory timing and triggering some rare race condition, using up more RAM than usual (or increasing the HDD utilization) and again triggering some conditions that wouldn't happen normally. You name it.

If it was me, I would try to run the binary on another machine and do a "clean room" recreation of the config file. That said, service without sources and missing configs without backups indicate severe organizational problems, so I would probably not want to work in such a place for a long time, unless I was hired by the CEO specifically to rectify things.

Re: Ask HN: Ever worked with a service that can never be restarted?

#138

Post by weego »

My experience in this situation is that the more you decide to the hero and take on things that everyone else is washing their hands of when you have no reason to above anyone else, the more likely it is that you'll also be shouldered with responsibility for things going wrong that were similarly nothing to do with you in the first place.

Prioritise a replacement and don't decide to try smart things with the running one.

Re: Ask HN: Ever worked with a service that can never be restarted?

#139

Post by duxup »

My first "real" job was working with some old equipment related to some big old IBM (and related) mainframes. To view or change anything you dialed in with a modem and then via a terminal application displayed and modified memory directly.

I sometimes have dreams about specific hex codes that were common / would mean bad things.

However, they were really good about documentation and updates / backups.

Best thing is to start working on folks in the business to develop a backup plan / make your voice heard about the potential fallout of a failure. Or even just ask "what would happen if" of the decision makers.

Re: Ask HN: Ever worked with a service that can never be restarted?

#140

Post by Piskvorrr »

zwp wrote: post #15
jomkr wrote: post #9

I'm not sure that 55 months of up-time indicates it's more or less likely to go down in the next month, but I'd guess more likely. Surely there are options, have you tried de-compiling the source from the binary? >but the physical config was accidentally overwritten and there are no backups Any old dev PCs lying around somewhere? It's worth reaching out to the old developers to see if they have a copy, in really old…

> reaching out to the old developers to see if they have a copy I think you mean "older devs that are still with the company" but I was once contacted by a former employer for exactly this: they had lost source code for the billing system. I didn't have the source code so that was easy to answer. Developers that have stolen intellectual property from former employers may want to consider carefully how they answer suc…

If you barge in, lawyers blazing, shouting about "stolen intellectual property", the likely response is "don't have any, never had", regardless of its veracity. OTOH, "disks that are left unerased by accident" _and_ generally being very open on the nature of the emergency is much more likely to bring help.
Post reply on HN