Mitogen, an infrastructure code baseline that sucks less

Cool. This is essentially about creating a single image cluster that runs python. I suppose if one can accept the need for python and a python module be installed on all "nodes" - this would start to look a lot like parallel python:

http://www.parallelpython.com/

From an architectural point of view, an si cluster could be viewed as code+communication (+data, but the data could be "wrapped" in code). Ssh and shell provides this - shell basically wraps libc and binary blobs as callable code (built-in + assumed-available code) - and if the cluster isn't too heterogeneous can be augmented with anything from c/go code (compiler assumed available) to machine code (scp /usr/bin/...).

And it is of course possible to build this on top of python. But the result can be much more than "just" for infrastructure - if it really works it becomes a transparent cluster platform.

With Kubernetes and other container schedulers gaining popularity and the host OS requiring less effort to be production ready, there is much less need for configuration management tooling. I don't see a lot of appeal for another tool in this space.

> given a remote machine and an SSH connection, just magically make Python code run on that machine, with no hacks involving error-prone shell snippets, temporary files, or hugely restrictive single use request-response shell pipelines, and suchlike.

I wonder if some of this functionality would be easier with a language that inherently supported distributing functions between nodes, e.g. Joe Armstrong's "favorite Erlang Program", the Universal Server:

https://joearms.github.io/published/2013-11-21-My-favorite-e...

I am missing a few of the points, re:

* how to bootstrap this onto a system with very limited assumptions about resources/access

* performance

* market effects of being able to re-use a bunch of existing libraries/scripts/tools written for configuration management or what-have-you in popular python

edit - and there's documentation for erlang over ssh:

* http://erlang.org/doc/apps/ssh/using_ssh.html

* http://erlang.org/pipermail/erlang-questions/2007-January/02...

* http://erlang.org/doc/man/slave.html

The appeal of Ansible is that you can declaratively spec infrastructure and it happens (more or less). I don't see how this competes with that.

Hi Simon,

The n-liner is:

    import subprocess
    import logging

    import mitogen.utils


    def main(router):
        # http://mitogen.readthedocs.io/en/latest/api.html#context-factories
        remote_host = router.ssh(hostname='myvps.mycorp.com')
        root = router.sudo(via=remote_host, username='root', password='god')

        rc = root.call(subprocess.check_call, 'ifconfig lo', shell=True)
        print 'Command return code was:', rc

    logging.basicConfig(level=logging.INFO)
    mitogen.utils.run_with_router(main)

Which gives me:

    INFO:mitogen.ctx.local.22050:stdout: lo        Link encap:Local Loopback
    INFO:mitogen.ctx.local.22050:stdout:           inet addr:127.0.0.1  Mask:255.0.0.0
    INFO:mitogen.ctx.local.22050:stdout:           inet6 addr: ::1/128 Scope:Host
    INFO:mitogen.ctx.local.22050:stdout:           UP LOOPBACK RUNNING  MTU:65536  Metric:1
    INFO:mitogen.ctx.local.22050:stdout:           RX packets:323421007 errors:0 dropped:0 overruns:0 frame:0
    INFO:mitogen.ctx.local.22050:stdout:           TX packets:323421007 errors:0 dropped:0 overruns:0 carrier:0
    INFO:mitogen.ctx.local.22050:stdout:           collisions:0 txqueuelen:1
    INFO:mitogen.ctx.local.22050:stdout:           RX bytes:109087018382 (101.5 GiB)  TX bytes:109087018382 (101.5 GiB)
    INFO:mitogen.ctx.local.22050:stdout:
    Command return code was: 0

Seems context name generation has broken :/ But it's running on the remote machine, you get the idea.

I'm working on a "mitop" example utility that implements a multi-host top replacement, with hierarchical statistical roll-ups, and documenting the implementation of that will be the basis for the Getting Started guide, but it depends on a bunch of stuff I'm not ready to work on (async connect), while there are still critical issues where hangs may still occur due due to an exception, and basic stuff like module preloading being incomplete (currently day 13 of working on that).

It will come eventually, but it will likely encourage many more bug reports than my hands can respond to once it is written, so it is kind of a feature that it is missing right now. :)