Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How the Python import system works

tenthousandmeters.com

Post reply on HN
Print view View original

171–180 of 209 posts

Re: How the Python import system works

#171

Post by rocqua »

blakesley wrote: post #164
rocqua wrote: post #139

Earlier quoted context omitted.

People have a script run with python, and want to use code in other files. This is not supported in python. For reasons beyond my understanding, you are supposed to put the script with python, or with the shebang, in a different directory. Alternatively, you can always use `python -m` to run your code.

> People have a script run with python, and want to use code in other files. This is not supported in python. Come again? This is what "import" does, obviously. This is not a clear explanation of your problem.

I missed one detail. The files and the script are in the same folder.

I say script here because the apparent intention is that scripts and modules are separate. That is why it's not easy to import functions from a file in the same directory as a script.

This restriction is very unexpected. And it is not borne out well by the fact that it is not obvious this distinction exists.

Re: How the Python import system works

#172

Post by eesmith »

rocqua wrote: post #137
robochat wrote: post #91

Earlier quoted context omitted.

Relative imports used to work much more naturally IMHO in python2 but then they broke it in python3 because Guido wanted scripts and modules to always be separate codebases. So, whereas it used to be easy to have a module that could also be run as a script inside a package, this is now very difficult to implement. To the extent that any python2 code that does this, should probably be refactored when being ported to p…

This decision has bit me in the ass so often. I want to organize my code logically in directories. As a script grows, I want the ability to spin out parts of that file to separate files. In order to do that in python, I need separate directories between the script and the spun-out functionality. This ends with a script that says "do function from module" and all code being in the module. Having code in different dire…

FWIW, my command-line tools and any support code are all in a subdirectory under my package directory.

I used to include a simple stub Python program (~10 lines) as a "script" in my setup.py that would import the right code and call it.

Then I learned that "entry_points" implemented most of the dispatch behavior I wanted.

I no longer have those scripts, just an entry that basically says "from abc.cli.prog1 import main; main()". The prog1.py, prog2.py, etc. look like normal command-line scripts, assuming the usual:

    if __name__ == "__main__":
        main()
The major downside is I can't suppress SIGPIPE and SIGINT until main() is called, leaving a wider window where something like ^C gives a unwanted Python stack trace.

Re: How the Python import system works

#173

Post by eesmith »

blakesley wrote: post #103
captainmuon wrote: post #45

There are two things that are annoying about Python's import system. Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...). Number two is, in order to do package man…

Earnest question: why are you all trying to use relative imports? What problem is that solving for you? I've never even bothered to try it out because it seems potentially problematic in the way all relative references can be, e.g., relative file paths.

While I rarely need it, I like being able to vendor a package by dropping it into a project subdirectory.

If the package uses absolute paths then I would need to tweak all the imports for the new absolute path.

Re: How the Python import system works

#174

Post by Izkata »

yongjik wrote: post #90

Probably half of the commenters here know this, but since we're here, this is my go-to boilerplate for starting a python script. (Probably won't work on Windows.) #!/bin/sh # Run the interpreter with -u so that stdout isn't buffered. "exec" "python3" "-u" "$0" "$@" import os import sys curdir = os.path.dirname(os.path.realpath(sys.argv[0])) # Add enough .. to point to the top-level project directory. sys.path.insert(…

Wouldn't this part:

    #!/bin/sh
    "exec" "python3" "-u" "$0" "$@"
Be the same as this?

    #!/usr/bin/env python3 -u

Re: How the Python import system works

#175

Post by globular-toast »

Izkata wrote: post #174
yongjik wrote: post #90

Probably half of the commenters here know this, but since we're here, this is my go-to boilerplate for starting a python script. (Probably won't work on Windows.) #!/bin/sh # Run the interpreter with -u so that stdout isn't buffered. "exec" "python3" "-u" "$0" "$@" import os import sys curdir = os.path.dirname(os.path.realpath(sys.argv[0])) # Add enough .. to point to the top-level project directory. sys.path.insert(…

Wouldn't this part: #!/bin/sh "exec" "python3" "-u" "$0" "$@" Be the same as this? #!/usr/bin/env python3 -u

One difference is that parameters in the shebang is non standard and not supported on all OSes. Linux supports it, though.

Re: How the Python import system works

#176

Post by formerly_proven »

rocqua wrote: post #171
blakesley wrote: post #164

Earlier quoted context omitted.

> People have a script run with python, and want to use code in other files. This is not supported in python. Come again? This is what "import" does, obviously. This is not a clear explanation of your problem.

I missed one detail. The files and the script are in the same folder. I say script here because the apparent intention is that scripts and modules are separate. That is why it's not easy to import functions from a file in the same directory as a script. This restriction is very unexpected. And it is not borne out well by the fact that it is not obvious this distinction exists.

When you run a script, the directory containing it is on the path. You can just write "from somefile import something". What you can't do is "from .somefile import something" because it is not a package.

When you run a module, the root package containing it is on the path by definition and imports work.

Re: How the Python import system works

#177

Post by Galanwe »

captainmuon wrote: post #45

There are two things that are annoying about Python's import system. Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...). Number two is, in order to do package man…

> Number two is, in order to do package management, you have to create a fake python installation and bend PYTHONPATH.

Well whatever the language or runtime, you need to tell it how to find its dependencies. Be it with a configuration file, environment variables or parameters.

Your statement is not exactly correct either. You don't need a fake installation _and_ bend PYTHONPATH.

Virtualenv leverages the fact that "../lib/python/site-packages/" relative to the interpreter is a default import path (default value of PYTHONHOME). It doesn't use PYTHONPATH.

> Why couldn't the import system resolve versions, too?

Not sure that would be really great. I prefer to have my dependencies all grouped together in a setup.py rather than scattered in various files at import time.

Re: How the Python import system works

#178

Post by rocqua »

formerly_proven wrote: post #176
rocqua wrote: post #171

Earlier quoted context omitted.

I missed one detail. The files and the script are in the same folder. I say script here because the apparent intention is that scripts and modules are separate. That is why it's not easy to import functions from a file in the same directory as a script. This restriction is very unexpected. And it is not borne out well by the fact that it is not obvious this distinction exists.

When you run a script, the directory containing it is on the path. You can just write "from somefile import something". What you can't do is "from .somefile import something" because it is not a package. When you run a module, the root package containing it is on the path by definition and imports work.

I didn't know that.

Does that still work if you call the script from a different directory (so `python bar/foo.py`)?

Imma guess it doesn't work with symlinked scripts. But I don't think it should without doing an install, in which case a module does make sense.

Re: How the Python import system works

#179

Post by Galanwe »

blakesley wrote: post #103
captainmuon wrote: post #45

There are two things that are annoying about Python's import system. Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...). Number two is, in order to do package man…

Earnest question: why are you all trying to use relative imports? What problem is that solving for you? I've never even bothered to try it out because it seems potentially problematic in the way all relative references can be, e.g., relative file paths.

I do use them quite often without much issue, not sure why some people are struggling with it.

Say you have a main package "mylib" with a subpackage "mylib.utils". Typically I like to see "mylib.utils" imports as being in one of (roughly) 4 categories:

- standard imports, that I would put first in the file (e.g. "import logging")

- external imports, that I would put second in the file (e.g. "import requests")

- library local imports, bits and pieces from the "mylib" package that you want to reuse in "mylib.utils", but are external to the current package (e.g. "from mylib.email import client")

- and package local imports, which I see as implementation details of the current subpackage, and should be agnostic from the overall architecture of "mylib" (e.g. "from .helpers import help_function")

The last category are modules that only make sense from within "mylib.utils", should relocate with it even if it is renamed or moved elsewhere, and shouldn't require a change whatever the structure of "mylib" becomes, which is why I would use "mylib.utils" relative imports in there.

Re: How the Python import system works

#180

Post by formerly_proven »

rocqua wrote: post #178
formerly_proven wrote: post #176

Earlier quoted context omitted.

When you run a script, the directory containing it is on the path. You can just write "from somefile import something". What you can't do is "from .somefile import something" because it is not a package. When you run a module, the root package containing it is on the path by definition and imports work.

I didn't know that. Does that still work if you call the script from a different directory (so `python bar/foo.py`)? Imma guess it doesn't work with symlinked scripts. But I don't think it should without doing an install, in which case a module does make sense.

The directory containing the file is added to the path, and the named file is imported as the __main__ module. That's why "if __name__ == '__main__':" works in script files.

I don't think Python will resolve symlinks here, so if that needs to work, you probably need to do it yourself before touching other code / files.

Post reply on HN