Surprised with the scenario of losing code for a binary that is still running. In which software domains is this common ?
Ask HN: Ever worked with a service that can never be restarted?
191–200 of 201 posts
Re: Ask HN: Ever worked with a service that can never be restarted?
#192In South Africa the citizens are facing a crises of such a service and its our electricity system. Predominately coal power plants, that are at end of life, and rolling blackouts is a common occurrence. It is necessary action, to avoid a blackout, given it would take weeks to get the system going again. Furthermore, high inequality and unemployment, makes it paramount to grow the economy, so these rolling blackouts h…
You guys need Jesus... and some solar panels.
Re: Ask HN: Ever worked with a service that can never be restarted?
#193Earlier quoted context omitted.
You can use /proc to get what you need. It has a memory map and the memory itself. Read the map, pull the heap segments out, and read them into files named after the addresses. On an identical machine, load the process up, attach via gdb, and (this is the tricky part) then mmap in the files saved from the other side at the right addresses. If you can get the roots into your heap (tough), you can use gdb'd python API…
Seems like a checkpoint restore process. Do you have experience of this or actual guide? I'm really curious as to the underlying mechanism.
Nothing so formal. The tricky parts are the non-memory program state: registers and instruction pointers for all threads.
Re: Ask HN: Ever worked with a service that can never be restarted?
#194My advice: 1. Suggest that the work to replace it is prioritized commensurate with the business impact caused whilst re-establishing service as if it went down right now 2. Remind them that it will go down at the worst possible time. 3. Ensure your name is attached to these two warnings. 4. Promise yourself you wouldn't run your business this way. 5. Get on with your life.
I would add to that: - Make a plan of exploratory steps that could trigger a failure, in order of increasing risks. - Then, if necessary, get a formal sign-off from someone with authority (perhaps a director of the company) to proceed with each step. Steps like that, which are usually very low risk but do risk taking down an unknown service, might include things like (just ideas from other comments): - Logging in to…
I read earlier that some people were putting legacy apps on VMs, but I didn't realize that you can transfer a full working system into a VM including memory... which is pretty much a full state snapshot. Awesome. This is this guy's solution right there, until a replacement is in place.
Re: Ask HN: Ever worked with a service that can never be restarted?
#195Unpopular advice: find a subtle way to make it crash, preferably stealthy, but if not possible, at least in a way that can be attributable to mild innocent incompetence instead of malice! Then there will be more and more interesting work to do for you and others, either rediscovering and properly documenting the config, or, hopefully, architecting and coding its replacement! In the aftermath, the organization will be…
Came here to post something similar.
Why jump through all these hoops to unfuck some inherently fucked system.
It's a bunch of work and time and effort to reverse engineer it just to get back to the point of still having a piece of shit legacy system.
Honestly this is effectively natural selection at play for businesses. They put themselves in this situation, if it dies and destroys the business ... good fucking riddance, the people who let this situation occur need to fail hard, it's the only way they will ever learn.
Just walk up to the machine and plug in a USB kill for 2 seconds, remove it, and get on with your life.
Re: Ask HN: Ever worked with a service that can never be restarted?
#196Earlier quoted context omitted.
It's possible to avoid losing a configuration.
It's possible, and it's possible to avoid losing source code, and its.possible to have valid backups, and keep your servers decemtly secure... yet it's sufficiently common (especially as the program ages) that its not trivially attributed to competency; rather you need continuous, unfailing competency, in the face of continuous environmental change (local changes, within the business, and external changes, as busines…
It's more than possible, it's expected.
While all of these things do happen, they are preventable and should not be treated as excusable, people need to be held accountable and systems / processes need to be put in place to ensure that these things don't reoccur.
Re: Ask HN: Ever worked with a service that can never be restarted?
#197"The physical config was accidentally overwritten and there are no backups". Welcome to legacy. So for a situation like this, there are several things that you need to think about. First and foremost... what is the impact when (not if) this process finally stops? This isn't just for technical people. You need a business impact assessment. You need the users involved. They're your lever for fighting the inevitable fea…
>"The physical config was accidentally overwritten and there are no backups". >Welcome to legacy. No. Not welcome to legacy. Welcome to a terrible company with terrible processes.
Fixing old, broken process is an interesting intellectual challenge in itself.
Re: Ask HN: Ever worked with a service that can never be restarted?
#198In South Africa the citizens are facing a crises of such a service and its our electricity system. Predominately coal power plants, that are at end of life, and rolling blackouts is a common occurrence. It is necessary action, to avoid a blackout, given it would take weeks to get the system going again. Furthermore, high inequality and unemployment, makes it paramount to grow the economy, so these rolling blackouts h…
You guys need Jesus... and some solar panels.
Unfortunately, the national power provider has a monopoly on energy, and is a political mine field. So independent power producers are ready to integrate massive solar and wind power capabilities, but due to politics they keep delaying the negotiations and contracts. Many private businesses are trying to produce their own power through solar and alike, but its expensive as that technology needs to be important and the rand is one of the most undervalued currencies.
Re: Ask HN: Ever worked with a service that can never be restarted?
#199If it is running you can still get at the binary file through /proc and recover it. The file system will only really delete a file when there are no more users and a running process counts as the file being in used (so that pages can be paged back in from it if needed).
> but the physical config was accidentally overwritten Overwritten, not deleted. Open file handles can't really help you there.
Yes, they can. The low level implementation is such that the inode will not be released. That's why securely overwriting a file is hard. Just opening a file for output and writing data is not enough, you need to open it for update and then write as much data as the old one had or more.
Re: Ask HN: Ever worked with a service that can never be restarted?
#200Memory dump will probably be what you want to do, but are you sure the process closed the file? If it's a config file, that's sensible the process has closed the file descriptor, but if it didn't (probably because of developer error), it's possible that file be still accessible (through inode for example in Linux). Actually I faced the same problem today; it was a virtual machine disk file that has been overwritten but virtual machine was running, so the (deleted) file was open and I could get it back.