Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Say “no” to import side‐effects in Python

chrismorgan.info

Post reply on HN
Print view View original

21–30 of 86 posts

Re: Say “no” to import side‐effects in Python

#21

Post by chrismorgan »

einhverfr wrote: post #16

I do most of my work with Perl, rather than Python, but in Perl there are several kinds of stock import-level side-effects which are actually quite helpful. These are pretty light-weight though. They boil down to: 1. Global lexical changes to Perl. Sometimes this is the whole point of an import (for example Carp::Always, which typically turns any warning or exception string into a full stack dump). I can't imagine do…

In Python, side-effects are simply not acceptable at all. One should instead put the code with the side-effect in a function and call it.

This is the approach taken by gevent, which allows you to replace the entire I/O stack. But importing it does not effect the change; you must explicitly call code to do that. This is done thus:

    from gevent.monkey import patch_all
    patch_all()
c.f. http://www.gevent.org/gevent.monkey.html

Re: Say “no” to import side‐effects in Python

#22

Post by Hello71 » 

    $ echo 3 > /proc/sys/vm/drop_caches
    $ time python2 -c 'help("modules")' >/dev/null
        /usr/lib64/python2.7/site-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion 'g_type_from_name (name) == 0' failed
      import gobject._gobject
    python2 -c 'help("modules")' > /dev/null  1.58s user 0.23s system 69% cpu 2.626 total
    $ time python3 -c 'help("modules")' > /dev/null
    python3 -c 'help("modules")' > /dev/null  2.00s user 0.17s system 74% cpu 2.928 total
    $ time python2 -c 'help("modules")' >/dev/null
    /usr/lib64/python2.7/site-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion 'g_type_from_name (name) == 0' failed
      import gobject._gobject
    python2 -c 'help("modules")' > /dev/null  1.26s user 0.11s system 99% cpu 1.375 total
    $ time python3 -c 'help("modules")' > /dev/null
    python3 -c 'help("modules")' > /dev/null  1.75s user 0.09s system 99% cpu 1.852 total
perhaps your issue is having too many things installed?

Re: Say “no” to import side‐effects in Python

#23

Post by chrismorgan »

Hello71 wrote: post #22

$ echo 3 > /proc/sys/vm/drop_caches $ time python2 -c 'help("modules")' >/dev/null /usr/lib64/python2.7/site-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion 'g_type_from_name (name) == 0' failed import gobject._gobject python2 -c 'help("modules")' > /dev/null 1.58s user 0.23s system 69% cpu 2.626 total $ time python3 -c 'help("modules")' > /dev/null python3 -c 'help("modules")' > /d…

The laptop I'm working on was decent when new, six years ago, but is now not the fastest thing on the block. It has certainly collected quite a lot of things there (mostly from system packages), but it is probably just one or two things that are spending most of the time (excluding I/O time).

It's interesting; now that I've tried running it a few more times, `help('modules')` on my Python 2.7 is getting down to six or so seconds. (On Python 3 it takes around 0.15 seconds.)

Re: Say “no” to import side‐effects in Python

#24

Post by Mithaldu »

chrismorgan wrote: post #21
einhverfr wrote: post #16

I do most of my work with Perl, rather than Python, but in Perl there are several kinds of stock import-level side-effects which are actually quite helpful. These are pretty light-weight though. They boil down to: 1. Global lexical changes to Perl. Sometimes this is the whole point of an import (for example Carp::Always, which typically turns any warning or exception string into a full stack dump). I can't imagine do…

In Python, side-effects are simply not acceptable at all. One should instead put the code with the side-effect in a function and call it. This is the approach taken by gevent, which allows you to replace the entire I/O stack. But importing it does not effect the change; you must explicitly call code to do that. This is done thus: from gevent.monkey import patch_all patch_all() c.f. http://www.gevent.org/gevent.monkey…

In Perl the act of simply importing that patch_all function would be a side-effect, as it's implemented as code in pure perl that changes the caller's symbol table. I suspect it would be fair to call your import up there also a side-effect, but that it slips the mind since it's implemented in the python core. :)

Re: Say “no” to import side‐effects in Python

#25

Post by ghfdaghkj »

chrismorgan wrote: post #15
kzrdude wrote: post #9

Qt and Gtk seem to be the problem here, and they won't change. They aren't python modules as much as they are application frameworks and assume to be in full control. Gtk calls sys.setdefaultencoding("utf-8") when imported, changing the way your strings behave in the whole process.

With regards to Qt, the problem is not Qt but rather that something else is actually trying to use Qt at a time when it shouldn't. pygtk setting the default encoding to UTF-8 is concerning—if true, that is certainly bad behaviour.

I haven't used GTK in a long time, but when I was using it, I was under the impression that pygtk was being deprecated, and everyone should be using GObject-Introspection instead.

Re: Say “no” to import side‐effects in Python

#26

Post by anon4 »

adamcharnock wrote: post #7

Can anyone suggest best practice alternatives to import side-effects?

Well, prohibiting all side-effects would be silly and if taken literally, you won't be able to even declare classes and functions, since these get compiled to bytecode that is essentially "instantiate a function object with this code blob", i.e. it's executable code with the side effect of binding a name in the module's scope to a new function object.

In general - do not do anything that might fail, might take a long time (>0.5 seconds), or can't be stopped easily.

Specifically - do not create windows, do not connect to the internet, do not try to create a database in a hardcoded location, do not connect to postgres. Do not do things that will require calls to the operating system other than allocating memory.

The one thing I'm willing to concede would be reading a default configuration file. But only if you're certain you've written it in a way that won't blow up if for whatever reason the default path is not readable for the current user, or in other edge cases.

If your library needs to talk to the OS, it will need to be passed initialization parameters. Just provide a top-level class that the user instantiates and passes all the needed initialization parameters. Resist the urge to do silly things in it like having a module-global instance variable for it that is set when the class is first instantiated and overloading the __new__ method to return that instance if it's not None.

Re: Say “no” to import side‐effects in Python

#27

Post by chrismorgan »

ghfdaghkj wrote: post #25
chrismorgan wrote: post #15

Earlier quoted context omitted.

With regards to Qt, the problem is not Qt but rather that something else is actually trying to use Qt at a time when it shouldn't. pygtk setting the default encoding to UTF-8 is concerning—if true, that is certainly bad behaviour.

I haven't used GTK in a long time, but when I was using it, I was under the impression that pygtk was being deprecated, and everyone should be using GObject-Introspection instead.

I dunno—I was making assumptions, not being very familiar with the GTK scene in Python.

Re: Say “no” to import side‐effects in Python

#28

Post by atilaneves »

I have the unfortunate "luck" to be using a Python library at work that _loves_ to use import side-effects. Importing it like God intended makes it parse command-line arguments and fail if it doesn't like what was passed in. And that's just the start. Nearly every __init__.py has code in it, including class definitions. I have no idea why.

Re: Say “no” to import side‐effects in Python

#29

Post by einhverfr »

Mithaldu wrote: post #24
chrismorgan wrote: post #21

Earlier quoted context omitted.

In Python, side-effects are simply not acceptable at all. One should instead put the code with the side-effect in a function and call it. This is the approach taken by gevent, which allows you to replace the entire I/O stack. But importing it does not effect the change; you must explicitly call code to do that. This is done thus: from gevent.monkey import patch_all patch_all() c.f. http://www.gevent.org/gevent.monkey…

In Perl the act of simply importing that patch_all function would be a side-effect, as it's implemented as code in pure perl that changes the caller's symbol table. I suspect it would be fair to call your import up there also a side-effect, but that it slips the mind since it's implemented in the python core. :)

Indeed. Also there are times (Moose for example) where you want to do more than import a function as written. For example Moo and Moose create a custom function "has" at import time that is them attached to the caller's namespace.

The reason for this is that you want to fully attach the function in the space as a native method and this requires some closures to make work sanely. In short you want:

   package foo;
   use Moose;
   has bar => (is => 'rw');
to behave identically as:

   package notfoo;
   use foo;
   foo::has(bar => (is=>'rw'));
If you rely on the caller the second example would add a bar accessor to the caller's namespace (notfoo) rather than to foo. In short you on't want to use "has" in the Moose namespace. You want to add a custom function to the importing namespace.

As I understand it you can't do this in Python not because it is a bad idea (it is a very good idea sometimes) but rather because of limitations in the language (no multiline lambdas, which you need in order to do useful stuff with the symbol table in this regard unless there is an equivalent in terms of defining a method the calling class inside a closure in the imported package). But again, my knowledge of Perl is much better than my knowledge of Python, so I could be wrong.

When you get into this sort of metaprogramming, custom symbol table manipulations are not only helpful but downright necessary. It is what allows you to add a sophisticated MOP when such is not included in core.

Post reply on HN