How the Python import system works • Hacker News Board

Re: How the Python import system works

#121

Post by aasasd » Sat, Jul 24, 2021, 8:53 PM UTC

Fordec wrote: post #51

sillysaurusx wrote: post #27

Fun fact: you can overload the Python import system to work with other languages that you create . I use this for my Python-based Lisp: https://github.com/shawwn/pymen/blob/ml/importer.py import foo checks for "foo.l", and if found, compiles it to foo.py on the fly and imports that instead. It's so cursed. I love it.

Wait, what? > that you create As in existing language supplied eg. Perl, Java, etc., or literally anything? Like bootstrapping your own home made language from scratch?

If the import system allows your code to run instead of the ‘import’ statement, and to produce the module however you want, then of course you can do whatever: load code from Google or StackOverflow results, if you wish.

Re: How the Python import system works

#122

Post by AdrianoKF » Sat, Jul 24, 2021, 8:55 PM UTC

One interesting use case of overwriting `builtins.__import__` I've encountered was the automatic hooking by ClearML [0] (experiment tracking, ...) into all sorts of common libraries like Matplotlib, Tensorflow, Pytorch, and friends.

The implementation is surprisingly straightforward, once you've come to terms with the basic idea, see [1] and the rest of the `clearml.binding` package.

[0]: https://clear.ml [1]: https://github.com/allegroai/clearml/blob/master/clearml/bin...

Re: How the Python import system works

#123

Post by sltkr » Sat, Jul 24, 2021, 8:55 PM UTC

matheusmoreira wrote: post #33

jvolkman wrote: post #23

My least favorite is that import ordering matters in some situations. Like if I just run "organize imports", all of a sudden dependency cycles pop up and everything is broken. Certainly a sign of things being misimported/misorganized, but stuff happens has systems grow fast. And solving these issues is always incredibly time consuming. Unless anyone knows of magical tools to help solve import issues?

> Unless anyone knows of magical tools to help solve import issues? Topological sorting? Always wondered why programming languages can't do what package managers do.

The breakage described can only occur if there are dependency cycles, so topological sorting can't fix it. If there are no cycles, then the order of imports doesn't matter.

edit: Actually I'm not even sure what kind of error we're talking about here. If two modules import each other and they both need access to the other's contents upon initialization, there is no ordering that will work. And if at most one needs access to the other, it will always work, no matter in which order they are imported. So I don't really know what the OP was talking about.

Re: How the Python import system works

#124

Post by amelius » Sat, Jul 24, 2021, 8:56 PM UTC

sillysaurusx wrote: post #27

Fun fact: you can overload the Python import system to work with other languages that you create . I use this for my Python-based Lisp: https://github.com/shawwn/pymen/blob/ml/importer.py import foo checks for "foo.l", and if found, compiles it to foo.py on the fly and imports that instead. It's so cursed. I love it.

You mean that you can override the import mechanism, which means that it allows you to do just about anything, including making it work with other languages.

That doesn't sound cursed to me, just flexible.

Re: How the Python import system works

#125

Post by kristjansson » Sat, Jul 24, 2021, 8:57 PM UTC

dheera wrote: post #79

I created this fun hack that taught me a LOT about the import system. Basically it allows you to import anything you want, even specify a version, and it will fetch it from PyPI live. Might be interesting to flesh this out in a way that's deployable. Basically instead of import tornado and hoping and praying that the user has read you README and pip installed the right version of tornado, you can do from magicimport…

Cool! But why not just throw the code a package with a version dependency on tornado?

Re: How the Python import system works

#126

Post by nyrikki » Sat, Jul 24, 2021, 9:03 PM UTC

Waterluvian wrote: post #111

brundolf wrote: post #32

Earlier quoted context omitted.

Circular imports in JS matter when the imported code is being called immediately at import time. If a file defines functions that later call functions from another file (and vice-versa), but those symbols will be populated before the function actually gets called, there's no problem

Yeah. It’s so flexible. I get frustrated at Python (Django) serializers that legitimately need to depend on each other. And the answer on the forums is to create a near duplicate class.

JavaScript does not provide namespaces by default which allows this.

Python is built upon namespaces and cycles in dependencies, either viewed as graph theory or kSAT introduce that fun NP-complete problem of version hell.

Using `need` in Javascript maintains the directed acyclic graph structure, but if you get fancy you will run into the same problems with circular depends in Python.

Karp's 21 and/or SAT will catch up with you at some point if you don't respect the constraints that make the problem tractable.

Note I am not saying I prefer or like pythons choices...but that they had to make one.

Re: How the Python import system works

#127

Post by joshuamorton » Sat, Jul 24, 2021, 9:08 PM UTC

brilee wrote: post #25

johncena33 wrote: post #3

I've been programming in Python professionally for more than five years. I consider myself quite a good programmer. I still don't have good grasp on Python's import system. Does anyone else have similar experience?

I'm a python readability approver at Google and I don't understand how the import system works

To be fair, Google's python avoids ~99% of the complexity of Python's import system by making all imports absolute and doing most things through blaze/bazel.

Re: How the Python import system works

#128

Post by cobbal » Sat, Jul 24, 2021, 9:08 PM UTC

amelius wrote: post #124

sillysaurusx wrote: post #27

Fun fact: you can overload the Python import system to work with other languages that you create . I use this for my Python-based Lisp: https://github.com/shawwn/pymen/blob/ml/importer.py import foo checks for "foo.l", and if found, compiles it to foo.py on the fly and imports that instead. It's so cursed. I love it.

You mean that you can override the import mechanism, which means that it allows you to do just about anything, including making it work with other languages. That doesn't sound cursed to me, just flexible.

What’s flexible for a dynamic scripting language is often cursed from a static perspective. Knowing what imports resolve to statically can be nice.

Re: How the Python import system works

#129

Post by dheera » Sat, Jul 24, 2021, 9:15 PM UTC

kristjansson wrote: post #125

dheera wrote: post #79

I created this fun hack that taught me a LOT about the import system. Basically it allows you to import anything you want, even specify a version, and it will fetch it from PyPI live. Might be interesting to flesh this out in a way that's deployable. Basically instead of import tornado and hoping and praying that the user has read you README and pip installed the right version of tornado, you can do from magicimport…

Cool! But why not just throw the code a package with a version dependency on tornado?

Because then you'd still have to install the package.

It's nice to have something that "just works" without having to install it. I like to call it Level 5 autonomous software -- software that can figure out how to run itself with zero complaints.

I actually use this for a lot of personal non-production scripts, I can just clone the script on any system and just run it, it will figure itself out.

Also, packages with version dependencies fuck up other packages with other version dependencies, unless you set up virtualenvs or dockers or condas for them, and those take additional steps.

magicimport.py uses virtualenv behind the scenes, so when it imports tornado 4.5 because some script wants 4.5, it won't mess up the 6.0 that's already on your system. In some sense it just automates the virtualenv and pip install process on-the-fly, to give the effect that the script "just works".

Re: How the Python import system works

#130

Post by rcfox » Sat, Jul 24, 2021, 9:19 PM UTC

captainmuon wrote: post #45

There are two things that are annoying about Python's import system. Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...). Number two is, in order to do package man…

> Why couldn't the import system resolve versions, too? You could say `import foo >= 1.0` and it would download it to some global cache, and the import the correct versions from the cache.

What do you do about conflicts? Or say you have `import foo >= 1.0` in a file, and `import foo == 2.4`, but the latest version is 2.5, so the first import grabbed the latest version, and you later realize you need 2.4?

Imagine running a report generator for 5 hours, only to have the formatting module require a conflicting version of something and erroring out at run time...