Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Ever worked with a service that can never be restarted?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 201 posts

Re: Ask HN: Ever worked with a service that can never be restarted?

#41

Post by sokoloff »

adrianmsmith wrote: post #25
Tilian wrote: post #4

The configuration still exists in memory somewhere, so you could extract it from a dump... whether you should want to do this is an entirely different question though.

That's not necessarily true I think. The program could read the config at startup, and create structures or objects inside its heap to represent the config. After that it would have no need for the actual bytes of the config.

Those heap objects are the configuration. Not the configuration input file, but the configuration itself.

It is then be a task to reverse engineer the coding file from the config data structures.

Re: Ask HN: Ever worked with a service that can never be restarted?

#42

Post by scandox »

arethuza wrote: post #27
scandox wrote: post #14

Earlier quoted context omitted.

> because the one thing that I learn as I learn more about computers is that they're basically working "by accident" So true. Thanks this is good advice. As it happens it is a physical server. I'm not actually mandated to do anything except replace it with a comparable service but this makes me feel like I'm racing against time!

Have you tried decompiling the executable to see if any sensible information can be gained that way?

Everyone's afraid to do anything on that machine at all. So even getting agreement to login or install software or run tcpdump or whatever is fraught.

Re: Ask HN: Ever worked with a service that can never be restarted?

#43

Post by viraptor »

barry-cotter wrote: post #22
jomkr wrote: post #9

I'm not sure that 55 months of up-time indicates it's more or less likely to go down in the next month, but I'd guess more likely. Surely there are options, have you tried de-compiling the source from the binary? >but the physical config was accidentally overwritten and there are no backups Any old dev PCs lying around somewhere? It's worth reaching out to the old developers to see if they have a copy, in really old…

> I'm not sure that 55 months of up-time indicates it's more or less likely to go down in the next month, but I'd guess more likely. Less likely https://en.wikipedia.org/wiki/Lindy_Effect > The Lindy effect is a theory that the future life expectancy of some non-perishable things like a technology or an idea is proportional to their current age, so that every additional period of survival implies a longer remaining l…

It's going to be some mix of Lindy and bathtub. Lindy for the software itself and electronics, bathtub for fans, spinning disks, cables, other things that may physically degrade.

Also, unless the server is in a data centre with some serious redundancies, (and even then...) the power is guaranteed to go down one day.

Re: Ask HN: Ever worked with a service that can never be restarted?

#44

Post by fxtentacle »

pjc50 wrote: post #21
dijit wrote: post #7

Actually, I have experience here. The problem is that anything you do that's potentially destructive in service of getting the system to be more sustainable is going to be met with heavy criticism. So you must be careful, the company has accepted the risk that they're in and you'll have to contend with that most likely. First thing is first: is it a VM or a Physical machine? Things get a little easier if it's a VM be…

Attaching a debugger potentially disrupts the service. I would suggest trying to live-image it with some other technique first.

Agree.

Attaching a debugger can cause multi-threaded processes to behave differently and/or deadlock if you are unlucky.

Also, some closed-source software components deliberately crash if they detect a debugger and since you don't have source code for the binary, there is a risk that one such component was used during development.

Re: Ask HN: Ever worked with a service that can never be restarted?

#45

Post by clarry »

barry-cotter wrote: post #22
jomkr wrote: post #9

I'm not sure that 55 months of up-time indicates it's more or less likely to go down in the next month, but I'd guess more likely. Surely there are options, have you tried de-compiling the source from the binary? >but the physical config was accidentally overwritten and there are no backups Any old dev PCs lying around somewhere? It's worth reaching out to the old developers to see if they have a copy, in really old…

> I'm not sure that 55 months of up-time indicates it's more or less likely to go down in the next month, but I'd guess more likely. Less likely https://en.wikipedia.org/wiki/Lindy_Effect > The Lindy effect is a theory that the future life expectancy of some non-perishable things like a technology or an idea is proportional to their current age, so that every additional period of survival implies a longer remaining l…

Ideas and technologies are actively sustained by external forces due to perceived merit in said ideas or technologies.

A computer program that is freewheeling along, by contrast, is not sustained by but dependent on external factors (software, hardware, electricity), all of which are prone to fail and some of which are more likely to fail with age.

Re: Ask HN: Ever worked with a service that can never be restarted?

#46

Post by faeyanpiraat »

ThomasRedstone wrote: post #40

A lot of talk about what to do yourself, I'd say do nothing yourself. Find an expert who has masses of experience who can consult on it. This isn't a good time to be learning and testing those lessons.

This is the solution in my experience aswell.

Doing even minor things that could distrupt the execution for any time (attaching debugger, installing packet sniffer that blocks network traffic for a sec or cause a packet lost, etc) could deadlock the application.

Wouldn’t risk it.

Re: Ask HN: Ever worked with a service that can never be restarted?

#47

Post by RandomBacon »

barry-cotter wrote: post #22
jomkr wrote: post #9

I'm not sure that 55 months of up-time indicates it's more or less likely to go down in the next month, but I'd guess more likely. Surely there are options, have you tried de-compiling the source from the binary? >but the physical config was accidentally overwritten and there are no backups Any old dev PCs lying around somewhere? It's worth reaching out to the old developers to see if they have a copy, in really old…

> I'm not sure that 55 months of up-time indicates it's more or less likely to go down in the next month, but I'd guess more likely. Less likely https://en.wikipedia.org/wiki/Lindy_Effect > The Lindy effect is a theory that the future life expectancy of some non-perishable things like a technology or an idea is proportional to their current age, so that every additional period of survival implies a longer remaining l…

OP is talking about a phyaical server. The Lindy Effect is for "some non-perishable things". A server that already has already been running for at least 55 monthes is definitely perishable.

Re: Ask HN: Ever worked with a service that can never be restarted?

#48

Post by rob74 »

jomkr wrote: post #9

I'm not sure that 55 months of up-time indicates it's more or less likely to go down in the next month, but I'd guess more likely. Surely there are options, have you tried de-compiling the source from the binary? >but the physical config was accidentally overwritten and there are no backups Any old dev PCs lying around somewhere? It's worth reaching out to the old developers to see if they have a copy, in really old…

Well, at least that's a plus for non-compiled languages (PHP, JS etc.): even if everything else fails, you still have the code on the server.

But I guess the bigger problem here is the missing config - as long as you still have the binary of the application, you should be able to restart it.

Re: Ask HN: Ever worked with a service that can never be restarted?

#49

Post by fxtentacle »

Hire a consultant to do a full-memory dump.

1. That gives you a pretty good chance of attaching a debugger (to the offline memory file) and extracting the config from memory without touching the running system.

2. You are safe in case things turn awful, which seems likely.

As for the source code, if it is an interpreted language like Java or Python or C#, your chances of recovering a fully-working source code tree are pretty good. At the very least, it'll help with understanding what the system does.

For C / C++, buy Hex-Rays IDA and Decompiler. That's what the pros use for reverse engineering video game protections, breaking DRM, etc. So that tool set is excellent for getting a overview of what a binary does with the ability to go into all the nitty-gritty details if you need help re-implementing one detail. Plus, Hex-Rays can actually propagate types and variable names through your disassembly and then export compile-able C source code for functions from your binary.

Re: Ask HN: Ever worked with a service that can never be restarted?

#50

Post by arethuza »

scandox wrote: post #42
arethuza wrote: post #27

Earlier quoted context omitted.

Have you tried decompiling the executable to see if any sensible information can be gained that way?

Everyone's afraid to do anything on that machine at all. So even getting agreement to login or install software or run tcpdump or whatever is fraught.

What I was meaning was that if you can get a hold of the binary that is running you could take it to some other environment and have a look (starting with a hex dump and working up from there...).

Mind you if things are that bad then maybe I'd follow my own advice (in another comment) and stay well away from the currently running thing.

Post reply on HN