Say “no” to import side‐effects in Python

Would anyone like to share their experiences avoiding this sort of problem in the context of web frameworks and building the back end for larger web sites/apps?

As an example for discussion, the first time I wrote a Flask-based back-end, I backed myself into a corner almost immediately in the following way.

Firstly, the WSGI file that the web server uses to start the application followed the suggestion in the Flask docs by doing this:

    # webserverseesthis.wsgi
    from yourapplication import app as application

That’s not so bad, but then I started doing application configuration and loading various Flask plug-ins as side effects of that import:

    # yourapplication/__init__.py
    app = Flask("yourapplication")
    
    # Do some general application configuration.
    app.config.from_pyfile("/path/to/configuration/file")

    # Set up some overarching security things that modify application behaviour.
    from flaskext.securityplugin import SecurityPlugin
    sp = SecurityPlugin(app)

This seemed at the time like the obvious place to put such things, but of course, this is really just a variation on the mistake we’re discussing here.

To compound the error, I then used Flask’s decorators to wire up routes from various URLs to the relevant parts of my code. Those decorators work on the application object (sticking with ideas common to many Python web frameworks and avoiding getting into anything more Flask-specific like blueprints) so I was effectively creating circular dependencies from almost everything to that top-level package:

    # yourapplication/pages/home.py
    from yourapplication import app

    @app.route('/')
    def home_page():
        # Render home page

and then from the top-level package onto almost everything so all those decorators could take effect:

    # After setting up the application object in yourapplication/__init__.py
    import yourapplication.pages.home

Now, as long as this kind of code only ever runs as a WSGI application behind a web server, you get away with these dependencies up to a point. In practice, your WSGI set-up imports the top-level application package, which in turn sets up the application object everything is going to depend on and only then imports all the supporting modules/packages, and everything “works”.

However, as soon as you want to write tests or otherwise reuse any of the code in a different context, the entire system is a big bowl of spaghetti with all the usual problems. The moment you import any part of the system to run a unit test on something in it, you get much of the rest of the system as well, complete with the side effects of any imports therein.

This was of course all horribly naïve on general programming principles, but the nature of these frameworks tends to push in this direction, and even Flask’s own documentation features various simple examples that follow a similar approach, so I’ll forgive myself for falling into the trap the first time. I’ve since experimented with various techniques to break the cycles and avoid the side effects on imports, with some success, but frankly I’ve never found a satisfying, general strategy for organising larger code bases built around a web framework.

How is everyone else doing this?

On one machine I tried, help('modules') actually worked successfully with no substantial delays or apparent side effects.

On another, it apparently tried to set up an MPI cluster:

  *** The MPI_Init() function was called before MPI_INIT was invoked.
  *** This is disallowed by the MPI standard.
  *** Your MPI job will now abort.
  [hostname:14114] Abort before MPI_INIT completed successfully; not 
  able to guarantee that all other processes were killed!

On a third, I get the following and then it just hangs:

  Python 2.7.6 (default, Mar 22 2014, 15:40:47)
  [GCC 4.8.2] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> help('modules')
  
  Please wait a moment while I gather a list of all available modules...

  /usr/lib/python2.7/dist-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion 'g_type_from_name (name) == 0' failed
    import gobject._gobject
  /usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: cannot register existing type 'GtkWidget'
    g_type = info.get_g_type()
  /usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: cannot add class private field to invalid type ''
    g_type = info.get_g_type()
  /usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: cannot add private field to invalid (non-instantiatable) type ''
    g_type = info.get_g_type()
  /usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: g_type_add_interface_static: assertion 'G_TYPE_IS_INSTANTIATABLE (instance_type)' failed
    g_type = info.get_g_type()
  /usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: cannot register existing type 'GtkBuildable'
    g_type = info.get_g_type()
  /usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: g_type_interface_add_prerequisite: assertion 'G_TYPE_IS_INTERFACE (interface_type)' failed
    g_type = info.get_g_type()
  /usr/lib/python2.7/dist-packages/gi/module.py:171: Warning: g_once_init_leave: assertion 'result != 0' failed
    g_type = info.get_g_type()

I'm on my mobile, so I can't paste code here. But the way I do it is to subclass the flask app and setup the routes in the setup method. Almost nothing happens on module scope. All views are defined in a seperate module using subclasses of flask views and they hold weak references to the app object. With this setup I neve had any problems testing individual modules. It also works together with flask's unit test client.

I only extensively used it from Perl and Vala, but with GObject-Introspection I tend to more often look at the original library documentation than something language specific. This can be a disadvantage at first, but for me it turned out more convenient, since GIR-inflated bindings can be more complete, as long as the inflation supports the features the API describes. They also tend to be more consistent in their differences to the original.

The problem about documentation seems to be the usual dilemma that as soon as you know enough to implement an API browser reading GIR that outputs the API in your language, you know enough about how the bindings work themselves to just use the original documentation.

I was referring more to Haskell's lack of side effects. I don't think you can do this in modules.