Live data from Hacker News

Ask HN: Ever worked with a service that can never be restarted?

news.ycombinator.com

141–150 of 201 posts

Re: Ask HN: Ever worked with a service that can never be restarted?

#141
post #134

Earlier quoted context omitted.

It's possible to avoid losing a configuration.

It's possible, and it's possible to avoid losing source code, and its.possible to have valid backups, and keep your servers decemtly secure... yet it's sufficiently common (especially as the program ages) that its not trivially attributed to competency; rather you need continuous, unfailing competency, in the face of continuous environmental change (local changes, within the business, and external changes, as busines…

> rather you need continuous, unfailing competency

That's called doing it manually.

What you're doing is making excuses rather than holding yourself to a higher level of quality.

Re: Ask HN: Ever worked with a service that can never be restarted?

#142
"The physical config was accidentally overwritten and there are no backups".

Welcome to legacy.

So for a situation like this, there are several things that you need to think about. First and foremost... what is the impact when (not if) this process finally stops? This isn't just for technical people. You need a business impact assessment. You need the users involved. They're your lever for fighting the inevitable fear-based political hurdles. Is it an annoyance? Or does the company go out of business? The potential severity of the impact matters a great deal. If it's putting the entire business at risk, you should be able to get support from the highest levels of management to do whatever is necessary.

Second... how do you recover? There are a variety of ways off the top of my head. The most obvious would be to reconstruct the physical config. The "obvious to others" that is probably a stupid idea is rewriting the application. Let's ignore the stupid one and start dealing with reconstructing the configuration.

Do you have the source code for the system? If so, you can probably reconstruct the configuration architecture from reading source, at least. It may suck, but it's something.

Is there a test environment with its own running copy of the app? If so, it will have its own configuration, which will make reconstruction much easier, as then you differ only by values and don't have to figure out what the fields are.

Now, what kind of data is in the configuration that makes it difficult? Resource locations? Authentication credentials? Something else? If it's connecting to external systems, you can look at logs, packet-sniff, etc, to at least figure out where it's going. Credentials can be reset for a new version - a painful one-way trip, but it can work. Do you own any external systems, or are they outside your control?

Now, all systems have inputs and outputs. What is the output of this? Is it going to a database? If so, are you backing up that data? Make sure any locally stored data is getting backed up!

If there isn't a duplicate test system, what would need to be done to create one? Are there licensing restrictions? Specialized hardware/OS? Are you building from source code? Do you have the source? Do whatever it takes to create a parallel system that you can test configuration on, make it run elsewhere.

I can just go on and on with this, but the important thing is to be able to duplicate as much as possible before you try to replace. And find out what the cost is - that buys you authority.

Re: Ask HN: Ever worked with a service that can never be restarted?

#144

Are you working on a Nuclear Reactor or something that will cause loss of life if rebooted accidentally? If yes, then you have a truly critical system that needs very careful uptime management, despite huge costs to carefully derisk and duplicate it, and there's plenty of good advice here already. But too many systems are 'super pets' like this and are mistakenly considered critical at exorbitant cost. If no: turn it…

Heh. Know of a team of engineers who deliberated over a brittle system for months. Mission critical, world ending stuff if it failed. Their manager got tired of waiting and sent a mail to the team that ran the data centre asking for the machine to be disconnected from the network. By CoB, nobody else had even noticed. Thanks to the nature of big businesses, that mission critical application had been programmed into obsolescence years before by other teams shifting their own workloads off of the server.

Re: Ask HN: Ever worked with a service that can never be restarted?

#145
post #15

Earlier quoted context omitted.

> reaching out to the old developers to see if they have a copy I think you mean "older devs that are still with the company" but I was once contacted by a former employer for exactly this: they had lost source code for the billing system. I didn't have the source code so that was easy to answer. Developers that have stolen intellectual property from former employers may want to consider carefully how they answer suc…

If you barge in, lawyers blazing, shouting about "stolen intellectual property", the likely response is "don't have any, never had", regardless of its veracity. OTOH, "disks that are left unerased by accident" _and_ generally being very open on the nature of the emergency is much more likely to bring help.

This case is one I'd be pretty careful about though as a former employee - the situation the company's found themselves in is pretty indicative of mismanagement, and desperate people do desperate things.

IANAL, but a hold harmless agreement would be a simple CYA that I'd do if I was personally in this situation.

Re: Ask HN: Ever worked with a service that can never be restarted?

#146
post #97

Earlier quoted context omitted.

What verytrivial said!!!! I’ve been in similar situations, but never without an immediately obvious solution. That system WILL FAIL. Even as we speak the time-to-failure is shrinking. Even if one of the solutions described below actually works, you won’t get 100% recovery. I recall a story years ago - from MIT, if memory serves - where they rebooted a system because they had many generations of Sybase backups. When t…

> That system WILL FAIL. I think that the key point. We have a system that can't easily be restarted and won't automatically start after a server is rebooted. The developers of the software don't care, because "what are the chances of a virtual machine spontaneously restarting". Turns out, those chances are rather good. Servers, virtual machines, containers, doesn't matter, unless that thing is running on a mainframe…

What's special about mainframes in this context?

Re: Ask HN: Ever worked with a service that can never be restarted?

#147
post #7

Actually, I have experience here. The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is going to be met with heavy criticism. So you must be careful, the company has accepted the risk that they're in and you'll have to contend with that most likely. First thing is first: is it a VM or a Physical machine? Things get a little easier if it's a VM be…

You can use /proc to get what you need. It has a memory map and the memory itself. Read the map, pull the heap segments out, and read them into files named after the addresses. On an identical machine, load the process up, attach via gdb, and (this is the tricky part) then mmap in the files saved from the other side at the right addresses. If you can get the roots into your heap (tough), you can use gdb'd python API…

Seems like a checkpoint restore process. Do you have experience of this or actual guide? I'm really curious as to the underlying mechanism.

Re: Ask HN: Ever worked with a service that can never be restarted?

#148

If it is running you can still get at the binary file through /proc and recover it. The file system will only really delete a file when there are no more users and a running process counts as the file being in used (so that pages can be paged back in from it if needed).

That depends on how the application works.

Most likely, the application just deserialized into its internal data structures and closed the file.

Re: Ask HN: Ever worked with a service that can never be restarted?

#149
My mother had a saying: Choose your battles. I feel like the value of calling things "not my problem" in software is undervalued. You cannot fix every problem, you cannot train every engineer, and you cannot control everything. Learn to prioritize.

If it is not on your head, don't fuck with it. I understand the instinct to fix a Big Problem and look great to management. However, this is too high risk. If you solve the problem, you are a hero. If you fuck it up, you are an idiot and fired and cost the company $$.

Why run the risk at all? Just cash the paychecks and fix other things that can't go catastrophically wrong.

Re: Ask HN: Ever worked with a service that can never be restarted?

#150
post #146

Earlier quoted context omitted.

> That system WILL FAIL. I think that the key point. We have a system that can't easily be restarted and won't automatically start after a server is rebooted. The developers of the software don't care, because "what are the chances of a virtual machine spontaneously restarting". Turns out, those chances are rather good. Servers, virtual machines, containers, doesn't matter, unless that thing is running on a mainframe…

What's special about mainframes in this context?

Mainframes tend to be designed with extreme uptime in mind - to the point of often having hot-swappable processors.
Post reply on HN