How the Python import system works • Hacker News Board

Re: How the Python import system works

#41

Post by simlevesque » Sat, Jul 24, 2021, 4:19 PM UTC

roenxi wrote: post #19

But despite being terrible the Python import system is remarkably easy to get started in. And generally easy for beginners to work with (put all the code in the same folder or "pip install"). There are some lessons here that other languages would do well to learn. Trouble importing 3rd party libraries must be a kiss of death for beginner engagement.

I disagree too... you can't even refer to a file in the same folder without using some magic (adding current folder to the path), which is a huge barrier when starting.

Re: How the Python import system works

#42

Post by toomuchtodo » Sat, Jul 24, 2021, 4:20 PM UTC

sillysaurusx wrote: post #27

Fun fact: you can overload the Python import system to work with other languages that you create . I use this for my Python-based Lisp: https://github.com/shawwn/pymen/blob/ml/importer.py import foo checks for "foo.l", and if found, compiles it to foo.py on the fly and imports that instead. It's so cursed. I love it.

A master of the dark arts I presume.

Re: How the Python import system works

#43

Post by chrisseaton » Sat, Jul 24, 2021, 4:21 PM UTC

btilly wrote: post #34

chrisseaton wrote: post #7

Earlier quoted context omitted.

> three modules cannot depend on each other in a circular way What would the purpose of circular modules like this be? You may as well collapse into a single module and the situation would not be any different would it?

What would the purpose of circular modules like this be? You may as well collapse into a single module and the situation would not be any different would it? What is the purpose of modules? You may as well collapse into a single script and the situation would not be any different, would it? I'm not being facetious here. The answer to the second is the answer to the first. A common example might go like this. You have…

> What is the purpose of modules?

So parts of the system can be managed independently.

> The answer to the second is the answer to the first.

Clearly not - since circular modules cannot be managed independently!

Re: How the Python import system works

#44

Post by josephorjoe » Sat, Jul 24, 2021, 4:23 PM UTC

abhisuri97 wrote: post #30

Despite using python for the past 4 years, it still takes me several tries to set up packages and imports correctly when I make them myself. Honestly, I wish that python had an import system similar to JS (where you can just say “I want this file” and specify the exports yourself). For me, it just feels more intuitive and less “magic”-like when dealing with custom scripts you want to import.

I have seen way too many `ModuleNotFoundError`s. It is moderately infuriating when the two files are in the same directory and python can't find the module.

Honestly that error is misnamed. It should be `ModuleImportRefusedError`.

And the frustration caused by getting PyTest to work in a project is likely responsible for a large percentage of the untested python projects in the world...

Re: How the Python import system works

#45

Post by captainmuon » Sat, Jul 24, 2021, 4:24 PM UTC

There are two things that are annoying about Python's import system.

Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...).

Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH. Virtualenvs are the canonical way to do it, but to me it feels like a hack - the core language seems to wants all packages installed in /usr. So now I have all these virtualenvs lying around and they are detached from the scripts.

Why couldn't the import system resolve versions, too? You could say `import foo >= 1.0` and it would download it to some global cache, and the import the correct versions from the cache.

Re: How the Python import system works

#46

Post by geofft » Sat, Jul 24, 2021, 4:26 PM UTC

fullstop wrote: post #21

johncena33 wrote: post #3

I've been programming in Python professionally for more than five years. I consider myself quite a good programmer. I still don't have good grasp on Python's import system. Does anyone else have similar experience?

I sometimes fall into the trap of using pip to install dependencies and then things break after an os update. That is, my python version has changed from 3.8 to 3.9 and my dependencies are sitting in the wrong directory. I never know if I should use pip and requirements.txt or rely on Ubuntu's packaged versions.

I used to have a rule of, never use pip and only use Ubuntu's/Debian's packaged versions. That works pretty well if you're happy with the packaged versions and you don't need unpackaged libraries.

I now have the rule of, only ever use pip inside a venv. If your venv is more than a little bit complex, write a requirements.txt file so you can generate it. So it's something like

    $ cat > requirements.txt  .gitignore
    $ python3 -m venv venv
    $ venv/bin/pip install -r requirements.txt
    $ venv/bin/python

or, if you prefer,

    $ . venv/bin/activate
    (venv)$ pip install -r requirements.txt
    (venv)$ python

Then when your Python version changes, or you get confused about what's installed, or whatever, you can just blow away the entire venv and recreate it:

    $ rm -r venv
    $ python3 -m venv venv
    $ venv/bin/pip install -r requirements.txt

and you're in a known-good place.

Either of these rules works fine. The thing that works poorly is using pip install outside of a venv (with or without root).

Re: How the Python import system works

#47

Post by Denvercoder9 » Sat, Jul 24, 2021, 4:30 PM UTC

captainmuon wrote: post #45

There are two things that are annoying about Python's import system. Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...). Number two is, in order to do package man…

> the core language seems to wants all packages installed in /usr.

There's also ~/.local/lib/python3/site-packages (or whatever your distribution made of that). Virtualenvs are only necessary if you want to isolate dependencies between environments. That's useful if you have projects with conflicting dependencies, because Python doesn't allow you to install multiple versions of the same package, for better or worse. However, if you've written some simple scripts that don't care much about the exact version of their dependencies, it's perfectly fine to install their dependencies glboally.

Re: How the Python import system works

#48

Post by th0ma5 » Sat, Jul 24, 2021, 4:31 PM UTC

sillysaurusx wrote: post #27

Fun fact: you can overload the Python import system to work with other languages that you create . I use this for my Python-based Lisp: https://github.com/shawwn/pymen/blob/ml/importer.py import foo checks for "foo.l", and if found, compiles it to foo.py on the fly and imports that instead. It's so cursed. I love it.

JPype does this for importing Java classes.

Re: How the Python import system works

#49

Post by kristaps » Sat, Jul 24, 2021, 4:31 PM UTC

captainmuon wrote: post #45

There are two things that are annoying about Python's import system. Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...). Number two is, in order to do package man…

Yeah, saner relative imports might be one of the few things that I envy the JS ecosystem for.

Re: How the Python import system works

#50

Post by CapmCrackaWaka » Sat, Jul 24, 2021, 4:35 PM UTC

GregJJ wrote: post #37

Python import system is by far the worst one I dealt with. Using Setup.py and regular or namespace packages, relative import, having complex sub packages and cross importing, running a script from somewhere inside one of your sub packages, and many more craps like these. Import system must be intuitive and easy to use!

Yeah it really tripped me up as a beginner. I think the hardest part to get used to was that the import syntax actually changes based on how, and from where, you are running your code. So depending on how you call your code, your imports might or might not work. This is ESPECIALLY painful when you are building a distribution. There is no syntax that works for all situations, which seems like it would be pretty important for an import system. I had to bookmark this tab, and still refer to it often.

https://stackoverflow.com/questions/14132789/relative-import...