How the Python import system works • Hacker News Board

Re: How the Python import system works

#201

Post by mixmastamyk » Tue, Jul 27, 2021, 1:52 AM UTC

fullstop wrote: post #21

johncena33 wrote: post #3

I've been programming in Python professionally for more than five years. I consider myself quite a good programmer. I still don't have good grasp on Python's import system. Does anyone else have similar experience?

I sometimes fall into the trap of using pip to install dependencies and then things break after an os update. That is, my python version has changed from 3.8 to 3.9 and my dependencies are sitting in the wrong directory. I never know if I should use pip and requirements.txt or rely on Ubuntu's packaged versions.

Just pip install again under new interpreter. Use distro versions if a script will primarily run there, and ok if an older version.

Re: How the Python import system works

#202

Post by mixmastamyk » Tue, Jul 27, 2021, 2:30 AM UTC

markmark wrote: post #150

rcxdude wrote: post #110

Earlier quoted context omitted.

It has some wildly frustratingly unintuitive behaviours in precisely the wrong place for beginners: in between having everything in a single script and building a proper package, especially when you are invoking your script with 'python script.py' as opposed to say 'python -m scripts.script'.

Yeah. Start writing a program in 'myprogram.py' as things grow do the right thing and split a function out to its own file and import it. It doesn't work. Suddenly you need to learn a whole bunch about python modules and the import system and scripts vs modules, and some of the questions you have just literally have no good answer.

Import by module:

    ⏵ ls
    module.py  script.py

    ⏵ cat module.py 
    def foo2():
        print('foo2 called.')
        
    ⏵ cat script.py 
    import module

    def foo1():
        print('foo1 called.')

    foo1()
    module.foo2()

    ⏵ python3 script.py
    foo1 called.
    foo2 called.

Import by name:

    ⏵ cat script.py 
    from module import foo2

    def foo1():
        print('foo1 called.')

    foo1()
    foo2()

    ⏵ python3 script.py
    foo1 called.
    foo2 called.

Re: How the Python import system works

#203

Post by mixmastamyk » Tue, Jul 27, 2021, 2:41 AM UTC

ASalazarMX wrote: post #74

hirowan wrote: post #58

I'm just grateful I'm not a core Python dev after reading this thread. I've never seen so much negativity concentrated in one place in quite some time, for a feature of a programming language which is fairly innocuous.

I love Python, but to be fair, the dependency/import system has not aged well, and the various projects trying to fix it are proof of that.

They're not trying to fix the import system.

Re: How the Python import system works

#204

Post by mixmastamyk » Tue, Jul 27, 2021, 2:42 AM UTC

trulyme wrote: post #96

grawprog wrote: post #81

Earlier quoted context omitted.

It was probably something I did. The original tutorial I followed had everything in one file and didn't get into anything about imports. I started splitting everything up arbitrarily and started tossing imports into the files that complained about missing dependencies and ended up getting overwhelmed because nothing worked. I'm sure if I'd taken the time to try and fix it I eventually could have and at this point i'v…

No, your impression was right. Reading this blog post made me realize how little I know about the python import system (and I use python daily), and at the same time how little I want to learn it. It is completely unintuitive and probably one of the worst aspects of otherwise beautiful and useful language. Fortunately, sys.path hack works reliably - one can just add that one line and imports work as expected.

There's only about three things to learn at first.

If you know how a path and relative path work from the shell, there is only one thing: touch __init__.py.

These are on the import doc page and beginners tutorial.

https://docs.python.org/3/tutorial/modules.html#packages

Re: How the Python import system works

#205

Post by mixmastamyk » Tue, Jul 27, 2021, 2:47 AM UTC

simlevesque wrote: post #41

roenxi wrote: post #19

But despite being terrible the Python import system is remarkably easy to get started in. And generally easy for beginners to work with (put all the code in the same folder or "pip install"). There are some lessons here that other languages would do well to learn. Trouble importing 3rd party libraries must be a kiss of death for beginner engagement.

I disagree too... you can't even refer to a file in the same folder without using some magic (adding current folder to the path), which is a huge barrier when starting.

Current folder is in the path by default.

Re: How the Python import system works

#206

Post by Doxin » Tue, Jul 27, 2021, 7:51 AM UTC

rocqua wrote: post #137

robochat wrote: post #91

Earlier quoted context omitted.

Relative imports used to work much more naturally IMHO in python2 but then they broke it in python3 because Guido wanted scripts and modules to always be separate codebases. So, whereas it used to be easy to have a module that could also be run as a script inside a package, this is now very difficult to implement. To the extent that any python2 code that does this, should probably be refactored when being ported to p…

This decision has bit me in the ass so often. I want to organize my code logically in directories. As a script grows, I want the ability to spin out parts of that file to separate files. In order to do that in python, I need separate directories between the script and the spun-out functionality. This ends with a script that says "do function from module" and all code being in the module. Having code in different dire…

So as your script grows, don't you eventually rename it to whatever/__main__.py and run it with -m? seems like a fairly trivial transform that then allows you to spin out as many modules as you need.

Re: How the Python import system works

#207

Post by grawprog » Tue, Jul 27, 2021, 5:03 PM UTC

mixmastamyk wrote: post #202

markmark wrote: post #150

Earlier quoted context omitted.

Yeah. Start writing a program in 'myprogram.py' as things grow do the right thing and split a function out to its own file and import it. It doesn't work. Suddenly you need to learn a whole bunch about python modules and the import system and scripts vs modules, and some of the questions you have just literally have no good answer.

Import by module: ⏵ ls module.py script.py ⏵ cat module.py def foo2(): print('foo2 called.') ⏵ cat script.py import module def foo1(): print('foo1 called.') foo1() module.foo2() ⏵ python3 script.py foo1 called. foo2 called. Import by name: ⏵ cat script.py from module import foo2 def foo1(): print('foo1 called.') foo1() foo2() ⏵ python3 script.py foo1 called. foo2 called.

>python3 script.py

I was using Python 2 at the time. Python 3 was still relatively new. Not sure how much difference it makes for your example, but the import systems are different between 2 and 3.

https://nerdparadise.com/programming/python/import2vs3

Re: How the Python import system works

#208

Post by josephorjoe » Wed, Jul 28, 2021, 4:27 PM UTC

mixmastamyk wrote: post #200

josephorjoe wrote: post #44

Earlier quoted context omitted.

I have seen way too many `ModuleNotFoundError`s. It is moderately infuriating when the two files are in the same directory and python can't find the module. Honestly that error is misnamed. It should be `ModuleImportRefusedError`. And the frustration caused by getting PyTest to work in a project is likely responsible for a large percentage of the untested python projects in the world...

I’ve not had either of these issues. Maybe a detail left out?

the first occurs if you haven't created __init__.py.

the second occurs if you want your test files outside of the package directory (e.g., project/src/foo_app/foo.py, project/test/test_foo.py)

Re: How the Python import system works

#209

Post by markmark » Mon, Aug 02, 2021, 10:14 AM UTC

mixmastamyk wrote: post #202

markmark wrote: post #150

Earlier quoted context omitted.

Yeah. Start writing a program in 'myprogram.py' as things grow do the right thing and split a function out to its own file and import it. It doesn't work. Suddenly you need to learn a whole bunch about python modules and the import system and scripts vs modules, and some of the questions you have just literally have no good answer.

Import by module: ⏵ ls module.py script.py ⏵ cat module.py def foo2(): print('foo2 called.') ⏵ cat script.py import module def foo1(): print('foo1 called.') foo1() module.foo2() ⏵ python3 script.py foo1 called. foo2 called. Import by name: ⏵ cat script.py from module import foo2 def foo1(): print('foo1 called.') foo1() foo2() ⏵ python3 script.py foo1 called. foo2 called.

I moved module.py into a modules folder to clean things up and now I get:

"ImportError: attempted relative import with no known parent package"

Looks like I'm back to having to learn a bunch of stuff about scripts and modules again?