Live data from Hacker News

How the Python import system works

tenthousandmeters.com

191–200 of 209 posts

Re: How the Python import system works

#191

Earlier quoted context omitted.

In JS land everything is a relative import. It's nice because you can move a whole directory of code from one project to another (or around in the same project) and it still works because all the imports pointing to other files inside that directory were relative, you only have to fix imports that go outside the directory. Also the way that JS imports are just relative paths is very nice because it means that the imp…

> It's nice because you can move a whole directory of code from one project to another (or around in the same project) and it still works Yes, just like absolute imports... > relative paths is very nice because it means that the imports are statically determinable, your editor can understand them and fix them automatically and you can trust that refactoring 1) Just like absolute imports? 2) I absolutely contest that…

In Python you can do (and people do do) this

    import a.b.c
    sys.path.insert(0, 'some/other/path')
    import x.y.z
Or

    env PYTHONPATH="foo/bar:$PYTHONPATH" python ...
There is no way in general for Python tooling to figure out where an import points. All it can do is guess. It's a very regrettable situation.

Yes in Java it's fine because Java has a build stage. So tooling can figure out where imports point to from your static configuration.

However from the tool's perspective there is nothing better than a relative path. Relative paths require no messing about with configuration files at all to resolve. It's just a path to another file or directory on disk.

When a tool sees

    import c from "./a/b/c"
It can resolve it immediately.

So what is the advantage of absolute imports really? If import lines are mainly written and maintained by tooling then shouldn't we pick the representation that is easiest for the tooling? Then we can have more and better tools and the tools will be more reliable.

And it turns out that, relative paths are easy for humans to understand too. The same configuration-free resolution algorithm also works in your head when you are reading code! At least when the language doesn't overcomplicate them too much (JS is guilty of this to a certain extent, although nowhere as bad as Python)

Re: How the Python import system works

#192
post #90

Probably half of the commenters here know this, but since we're here, this is my go-to boilerplate for starting a python script. (Probably won't work on Windows.) #!/bin/sh # Run the interpreter with -u so that stdout isn't buffered. "exec" "python3" "-u" "$0" "$@" import os import sys curdir = os.path.dirname(os.path.realpath(sys.argv[0])) # Add enough .. to point to the top-level project directory. sys.path.insert(…

> # Add enough .. to point to the top-level project directory. This suggests that there is more than one entry point to the Python project? While I'm sure there are good reasons for this, and while I'm not criticising your instance of this specifically, as a general point of advice I've found this sort of thing to be a bit of an anti-pattern. Having one entry that handles things like path setup and other global conce…

I use a very similar flow. The highly-opinionated-yet-effective pattern I use involves pydantic, cleo, and entry_points/console_scripts in setup.py.

- everything is structured as a module

- options and args are stored in their own module for easy reuse

- the whole stack has one cleo.Application, with however many subcommands. Usually of the form "mytool domain verb" e.g. "mytool backend start."

- cleo args/options are parsed into pydantic objects for automatic validation (you could do this with argparse and dataclasses to skip the deps but it's more work)

- each subcommand has a `main(args: CliArgsModel)` which takes that parsed structure and does its thing. This makes it super easy to unit test

I install into a venv with `poetry install` or `pip install -e` for editable installs.

It all just works, no fuss, and so damn modular.

Re: How the Python import system works

#193

Fun fact: you can overload the Python import system to work with other languages that you create . I use this for my Python-based Lisp: https://github.com/shawwn/pymen/blob/ml/importer.py import foo checks for "foo.l", and if found, compiles it to foo.py on the fly and imports that instead. It's so cursed. I love it.

Lots of things you can do with Python but probably shouldn't and people typically don't. That's one reason I prefer it to Ruby, or even Node, where monkey-patching or otherwise exposing bad magical behaviours is common and even encouraged -- the power is all there, but the ecosystem encourages you to use it for good, not evil. This sounds very much like the good kind of magic, though.

I don't even know what that means.

Re: How the Python import system works

#194

Earlier quoted context omitted.

> three modules cannot depend on each other in a circular way What would the purpose of circular modules like this be? You may as well collapse into a single module and the situation would not be any different would it?

The purpose is you want to use code in other modules. If you keep doing that for a while, a circular dependency will happen. This is the dumbest thing thing in Python. All other languages I know have solved it.

I've never had a circular dependency I couldn't resolve. Just organize your modules into a DAG. I have a pretty standard flow:

- protocols and abcs. Depend only on builtins, this sits at the top of the hierarchy and can be used anywhere

- top-level types/models, data structures. Only depends on external libraries

- config and logging. Depends on pydantic, which handles all init config validation, sets up log level. Many things import this

- specific class implementations, pure functions, transforms. imports from all the above

- dependency-injected stuff

- application logic

- routers and endpoints

I've had some close calls since I type check heavily, but protocols (since 3...6? 3.7 for sure) are a fantastic tool for both structuring and appeasing mypy.

Re: How the Python import system works

#195
post #186

Earlier quoted context omitted.

I disagree. I find it hard to get started in python. There are loads of package managers, so I don't know which one to pick. There are multiple different rules for for how imports work with local files. The standard library is full of of functions that one should not use any more, and you need to know which is which. Defining a main entrypoint consists of checking a magic __NAME__ constant. You have internalised all…

Counterpoints: Start with pip, the official package manager. You don't have to start with virtual envs: I didn't. I haven't ran into any function which I "should not use". You don't need an entry point, you can just write code in a file. Besides that, I don't see how much it differs from other languages with implicit entry points, where you need to match a certain name in your function. Imports: yes, those are annoyi…

Honestly I have had very few issues with poetry, and the ones I've had, it's because I'm trying to use plugins, which are alpha right now.

Dependency resolution just works. Editable installs just work. Building just works.

Before that, I only had problems with virtual envs once, and it was due to bad hygiene with system python libs and deps. Moral of the story: don't. Unless it's necessary to bootstrap virtualenv or compiled libs, don't system install python. Good ol' get-pip.py and virtualenv.

Re: How the Python import system works

#196

I've been programming in Python professionally for more than five years. I consider myself quite a good programmer. I still don't have good grasp on Python's import system. Does anyone else have similar experience?

I have >10y python exp. Vanilla import system? I grasp it quite well.

- Entrypoints? Getting there.

- Namespace packages? Ehh. Murky

- site-packages/mypackage.pth - I get it but I don't know why sometimes it appears and other times not

- c extensions? Ehhh.

- .so loading? Kinda magic.

- the confluence of editable installs, namespace packages, foo.pth, PYTHONPATH, sys.path, relative imports, entry points, virtualenvs, LD_LIBRARY_PATH, PATH, shell initialization, setup.py vs pyproject.toml? Um yeah that's some heavy wizardry.

Tbf, you don't need the vast majority of that to be effective in python.

Re: How the Python import system works

#198
post #89

One of the reasons I gradually fell out of love with Python. To get Python right you need to remember more protocol than Queen Victoria's master of tea. And it is truly protocol in the sense that there is always this arbitrariness hanging around it.

What do you use now?

Python, hehe. But I just don't like it that much anymore.

Re: How the Python import system works

#199

It's certainly a lot better than Ruby's require which just executes code and alters global virtual machine state. Not too different from C's #include. My favorite is Javascript's. Modules are objects containing functions and data, require returns such objects. Simple, elegant. Completely reifies the behind-the-scenes complexity of Python's import, making it easily understandable.

Though that's not how JS's new module system works, which I would say is also elegant in its own ways, but juggling two different systems is less so

I have no idea why they added a module system to the Javascript language itself. The old one was so awesome. Not sure what advantages the new system offers.

Re: How the Python import system works

#200

Despite using python for the past 4 years, it still takes me several tries to set up packages and imports correctly when I make them myself. Honestly, I wish that python had an import system similar to JS (where you can just say “I want this file” and specify the exports yourself). For me, it just feels more intuitive and less “magic”-like when dealing with custom scripts you want to import.

I have seen way too many `ModuleNotFoundError`s. It is moderately infuriating when the two files are in the same directory and python can't find the module. Honestly that error is misnamed. It should be `ModuleImportRefusedError`. And the frustration caused by getting PyTest to work in a project is likely responsible for a large percentage of the untested python projects in the world...

I’ve not had either of these issues. Maybe a detail left out?
Post reply on HN