How the Python import system works • Hacker News Board

Re: How the Python import system works

#51

Post by Fordec » Sat, Jul 24, 2021, 4:39 PM UTC

sillysaurusx wrote: post #27

Fun fact: you can overload the Python import system to work with other languages that you create . I use this for my Python-based Lisp: https://github.com/shawwn/pymen/blob/ml/importer.py import foo checks for "foo.l", and if found, compiles it to foo.py on the fly and imports that instead. It's so cursed. I love it.

Wait, what?

> that you create

As in existing language supplied eg. Perl, Java, etc., or literally anything? Like bootstrapping your own home made language from scratch?

Re: How the Python import system works

#52

Post by tyingq » Sat, Jul 24, 2021, 4:49 PM UTC

There's a very cool (and succinct) blog post[1] showing how to abuse this in an interesting way where you can put the code of a module into a string and load it that way.

Not something I'd use in production, but it's a very clear way to see how both "finding a module" and "loading a module" works under the covers.

[1] https://cprohm.de/blog/python-packages-in-a-single-file/

Edit: As an aside, I much prefer the way Perl does things in this space. It's much easier to define multiple packages either within a single file or across different files, and much clearer about what's happening underneath.

Re: How the Python import system works

#53

Post by [deleted] » Sat, Jul 24, 2021, 5:00 PM UTC

btilly wrote: post #34

chrisseaton wrote: post #7

Earlier quoted context omitted.

> three modules cannot depend on each other in a circular way What would the purpose of circular modules like this be? You may as well collapse into a single module and the situation would not be any different would it?

What would the purpose of circular modules like this be? You may as well collapse into a single module and the situation would not be any different would it? What is the purpose of modules? You may as well collapse into a single script and the situation would not be any different, would it? I'm not being facetious here. The answer to the second is the answer to the first. A common example might go like this. You have…

[deleted]

Re: How the Python import system works

#54

Post by lp251 » Sat, Jul 24, 2021, 5:01 PM UTC

chrisseaton wrote: post #7

MeteorMarc wrote: post #4

Python importing quirks can be time consuming. Things beginners will encounter: - three modules cannot depend on each other in a circular way - relative imports are fragile ("module not found") - the __all__ definitions in the __init__ file make modules available under different full names - how to reload a module in a jupyter notebook if edited and so on.

> three modules cannot depend on each other in a circular way What would the purpose of circular modules like this be? You may as well collapse into a single module and the situation would not be any different would it?

Suppose you have two classes, A and B. They are sufficiently complex to merit their own modules.

Suppose you have some method of A which does something special if it gets an instance of B, and vice versa. Now you have a circular import problem; glhf

Re: How the Python import system works

#55

Post by chrisseaton » Sat, Jul 24, 2021, 5:03 PM UTC

lp251 wrote: post #54

chrisseaton wrote: post #7

Earlier quoted context omitted.

> three modules cannot depend on each other in a circular way What would the purpose of circular modules like this be? You may as well collapse into a single module and the situation would not be any different would it?

Suppose you have two classes, A and B. They are sufficiently complex to merit their own modules. Suppose you have some method of A which does something special if it gets an instance of B, and vice versa. Now you have a circular import problem; glhf

> Now you have a circular import problem; glhf

But why not collapse into a single module at this point if you can’t avoid dependency? What are the separate modules adding at this stage forward?

Re: How the Python import system works

#56

Post by colonwqbang » Sat, Jul 24, 2021, 5:03 PM UTC

jvolkman wrote: post #23

My least favorite is that import ordering matters in some situations. Like if I just run "organize imports", all of a sudden dependency cycles pop up and everything is broken. Certainly a sign of things being misimported/misorganized, but stuff happens has systems grow fast. And solving these issues is always incredibly time consuming. Unless anyone knows of magical tools to help solve import issues?

Yes, it is unfortunate. Loading modules can have side effects as the loaded module is allowed to execute arbitrary code at load time. This is also a source of ordering issues.

Maybe some think this is only a theoretical problem and doesn't happen with "well-written" libraries. Well, here is one example which bit me in the past: https://stackoverflow.com/a/4706614/767442

Re: How the Python import system works

#57

Post by chriswarbo » Sat, Jul 24, 2021, 5:06 PM UTC

Denvercoder9 wrote: post #47

captainmuon wrote: post #45

There are two things that are annoying about Python's import system. Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...). Number two is, in order to do package man…

> the core language seems to wants all packages installed in /usr. There's also ~/.local/lib/python3/site-packages (or whatever your distribution made of that). Virtualenvs are only necessary if you want to isolate dependencies between environments. That's useful if you have projects with conflicting dependencies, because Python doesn't allow you to install multiple versions of the same package, for better or worse.…

Quick shout-out to nix-shell shebangs, which allow a script to specify exact versions of all dependencies, Python or otherwise, which will be cached into a temporary sandbox.

https://nixos.org/manual/nix/stable/#use-as-a-interpreter

I wrote a Python script yesterday which calls out to a couple of external commands (`mlr` and `xq`), with a shebang like this:

    #!/usr/bin/env nix-shell
    #!nix-shell -i python3 -p python3 -p miller -p yq

Re: How the Python import system works

#58

Post by hirowan » Sat, Jul 24, 2021, 5:10 PM UTC

I'm just grateful I'm not a core Python dev after reading this thread. I've never seen so much negativity concentrated in one place in quite some time, for a feature of a programming language which is fairly innocuous.

Re: How the Python import system works

#59

Post by lp251 » Sat, Jul 24, 2021, 5:10 PM UTC

chrisseaton wrote: post #55

lp251 wrote: post #54

Earlier quoted context omitted.

Suppose you have two classes, A and B. They are sufficiently complex to merit their own modules. Suppose you have some method of A which does something special if it gets an instance of B, and vice versa. Now you have a circular import problem; glhf

> Now you have a circular import problem; glhf But why not collapse into a single module at this point if you can’t avoid dependency? What are the separate modules adding at this stage forward?

You can, but sometimes it’s not ideal.

I ran into this when A and B had many derived classes. I wanted to put A and it’s derived classes in one module, and B and it’s derived classes in another. It was messy.

I wound up putting A and B in a single module and having a separate one for the derived classes. Not ideal.

Re: How the Python import system works

#60

Post by 0xC0ncord » Sat, Jul 24, 2021, 5:11 PM UTC

lp251 wrote: post #54

chrisseaton wrote: post #7

Earlier quoted context omitted.

> three modules cannot depend on each other in a circular way What would the purpose of circular modules like this be? You may as well collapse into a single module and the situation would not be any different would it?

Suppose you have two classes, A and B. They are sufficiently complex to merit their own modules. Suppose you have some method of A which does something special if it gets an instance of B, and vice versa. Now you have a circular import problem; glhf

I generally solve this problem by having a module specifically containing the abstract base classes of each of the classes I will be working with that implements either no or bare minimum functionality for these objects. That way, any other module can import this module and have visibility of every other class I will be working with.