Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How the Python import system works

tenthousandmeters.com

Post reply on HN
Print view View original

141–150 of 209 posts

Re: How the Python import system works

#141

Post by zarzavat »

scaryclam wrote: post #132
blakesley wrote: post #103

Earlier quoted context omitted.

Earnest question: why are you all trying to use relative imports? What problem is that solving for you? I've never even bothered to try it out because it seems potentially problematic in the way all relative references can be, e.g., relative file paths.

So much this. There's no good reason to use relative imports. They're less readable, more dangerous and don't solve a single problem.

In JS land everything is a relative import. It's nice because you can move a whole directory of code from one project to another (or around in the same project) and it still works because all the imports pointing to other files inside that directory were relative, you only have to fix imports that go outside the directory.

Also the way that JS imports are just relative paths is very nice because it means that the imports are statically determinable, your editor can understand them and fix them automatically and you can trust that refactoring. Python has turing complete imports because there's so much dynamic messing about with sys.path that goes on in Python due to inadequacies of the import system.

Re: How the Python import system works

#142

Post by BadInformatics »

Waterluvian wrote: post #10
MeteorMarc wrote: post #4

Python importing quirks can be time consuming. Things beginners will encounter: - three modules cannot depend on each other in a circular way - relative imports are fragile ("module not found") - the __all__ definitions in the __init__ file make modules available under different full names - how to reload a module in a jupyter notebook if edited and so on.

I’m always amazed just how tolerant javascript’s import system is when I have circular imports. I guess maybe because it doesn’t care about modules and just cares about specific elements that are being imported/exported. When I do have a nasty circular dependency Webpack usually does a bad job telling me what’s wrong. Though I should still treat circular imports as, at the very least, an organization code smell.

IIRC an explicit design goal was to enable circular deps (hence why imported bindings are considered "live". It's interesting to see this works in practice though, I've never tried using them myself.

Re: How the Python import system works

#143

Post by btilly »

dragonwriter wrote: post #134
btilly wrote: post #34

Earlier quoted context omitted.

What would the purpose of circular modules like this be? You may as well collapse into a single module and the situation would not be any different would it? What is the purpose of modules? You may as well collapse into a single script and the situation would not be any different, would it? I'm not being facetious here. The answer to the second is the answer to the first. A common example might go like this. You have…

> But you just created a circular dependency! Only if those things not only need to be able to “get to” each other, but also need to know, at compile time, about the concrete implementation of the others. That's possible to be a real need, but its also something that often happens because of excessive and unnecessary coupling.

The coupling that I described needs to be in the software because it exists in the real world that the software is trying to describe.

However your "compile time" point is important. There is another solution, which is to implement lazy loading of those classes.

So you put your import in the method of each that needs to know the other. This breaks the circular dependency and needs more up front memory. However it can also become a maintenance issue where a forgotten import in one function is masked by a successful import in another, until something changes the call and previously working code mysteriously goes boom.

It's all tradeoffs.

Re: How the Python import system works

#144

Post by yongjik »

dagmx wrote: post #136
yongjik wrote: post #90

Probably half of the commenters here know this, but since we're here, this is my go-to boilerplate for starting a python script. (Probably won't work on Windows.) #!/bin/sh # Run the interpreter with -u so that stdout isn't buffered. "exec" "python3" "-u" "$0" "$@" import os import sys curdir = os.path.dirname(os.path.realpath(sys.argv[0])) # Add enough .. to point to the top-level project directory. sys.path.insert(…

Why not just add a shebang , chmod +x and then you're done?

    #!/usr/bin/python
This is sometimes what you want, but it will always look at this exact path, and won't play nicely with virtualenv/conda.

    #!/usr/bin/env python
This works - it will use Python found in $PATH. Unfortunately you can't add any more parameters to the interpreter.

The contraption I wrote allows adding arbitrary parameters - I was burnt one too many times by Python silently buffering my debug messages, so I use it to always add "-u".

Re: How the Python import system works

#145

Post by btilly »

wpietri wrote: post #69
chrisseaton wrote: post #43

Earlier quoted context omitted.

> What is the purpose of modules? So parts of the system can be managed independently. > The answer to the second is the answer to the first. Clearly not - since circular modules cannot be managed independently!

To me the purpose of models is to help humans manage code. Our brains don't hold much at once, so the more we can forget about in a given circumstance, the easier it is. So I think btilly is correct: the reason I want modules, ignoring things I don't care about, is the same reason I want them to deal reasonably with circular references.

Exactly.

In the example that I gave, the design described will handle complex edge cases such as a part time employee working for multiple companies. And will do so without programmers having to think through the whole system at all points.

Independence of modules has no importance in a codebase that ships as a whole. But modularity does.

Re: How the Python import system works

#146

Post by zarzavat »

Waterluvian wrote: post #10
MeteorMarc wrote: post #4

Python importing quirks can be time consuming. Things beginners will encounter: - three modules cannot depend on each other in a circular way - relative imports are fragile ("module not found") - the __all__ definitions in the __init__ file make modules available under different full names - how to reload a module in a jupyter notebook if edited and so on.

I’m always amazed just how tolerant javascript’s import system is when I have circular imports. I guess maybe because it doesn’t care about modules and just cares about specific elements that are being imported/exported. When I do have a nasty circular dependency Webpack usually does a bad job telling me what’s wrong. Though I should still treat circular imports as, at the very least, an organization code smell.

Circular imports in JS are fine and not a code smell, for instance in typescript you may have two classes that reference each other's types - obviously this is not a real import from JS's perspective but the point is that you should not have to care whether it is real or not. That's a world we don't want to live in.

Circular imports are only ever a problem when you have code running when the module loads. Then you run into module load ordering issues. So avoid any side effects on module load and make all setup explicit.

Re: How the Python import system works

#147

Post by noptd »

zarzavat wrote: post #141
scaryclam wrote: post #132

Earlier quoted context omitted.

So much this. There's no good reason to use relative imports. They're less readable, more dangerous and don't solve a single problem.

In JS land everything is a relative import. It's nice because you can move a whole directory of code from one project to another (or around in the same project) and it still works because all the imports pointing to other files inside that directory were relative, you only have to fix imports that go outside the directory. Also the way that JS imports are just relative paths is very nice because it means that the imp…

Given most editors handle absolute imports as good as if not better than relative imports, I don't see any real benefit to using the latter.

Re: How the Python import system works

#148

Post by codethief »

xavdid wrote: post #85
captainmuon wrote: post #45

There are two things that are annoying about Python's import system. Number one is that relative imports are weird. My intuition about imports is good enough that I never bothered to learn all the rules explicitly, but sometimes something simple is just not possible and it bites me. I think the case is importing files relative to a script (and not running with python -m ...). Number two is, in order to do package man…

Once I wrapped my head around when you can and can't use relative imports, I've been pretty ok with them. The think that irks me is whether they work changes based on where you've invoked Python from. `./bin/my_script.py` behaves differently from `./my_script.py`. Coming from JS, that was a pretty frustrating realization.

> The think that irks me is whether they work changes based on where you've invoked Python from. `./bin/my_script.py` behaves differently from `./my_script.py`.

This.

IMO that's also one of the main issues of Bash – you can't modularize a script unless you make sure your working directory is the directory containing the script. (And good luck with finding out the latter! [0])

[0]: https://stackoverflow.com/questions/4774054/reliable-way-for...

Re: How the Python import system works

#149

Post by codethief »

danpalmer wrote: post #140
yongjik wrote: post #90

Probably half of the commenters here know this, but since we're here, this is my go-to boilerplate for starting a python script. (Probably won't work on Windows.) #!/bin/sh # Run the interpreter with -u so that stdout isn't buffered. "exec" "python3" "-u" "$0" "$@" import os import sys curdir = os.path.dirname(os.path.realpath(sys.argv[0])) # Add enough .. to point to the top-level project directory. sys.path.insert(…

> # Add enough .. to point to the top-level project directory. This suggests that there is more than one entry point to the Python project? While I'm sure there are good reasons for this, and while I'm not criticising your instance of this specifically, as a general point of advice I've found this sort of thing to be a bit of an anti-pattern. Having one entry that handles things like path setup and other global conce…

I have been a huge fan of this issue, too!

For additional bonus points, have your single entry point exhibit a CLI that properly documents everything the developer can do, i.e. what features are available, what environment variables and config flags can be set etc. That way, the code essentially documents itself and you know longer have to keep your README file updated (which people always tend to forget).

Re: How the Python import system works

#150

Post by markmark »

rcxdude wrote: post #110
toxik wrote: post #72

Earlier quoted context omitted.

Strange, Python’s import is not very difficult to use.

It has some wildly frustratingly unintuitive behaviours in precisely the wrong place for beginners: in between having everything in a single script and building a proper package, especially when you are invoking your script with 'python script.py' as opposed to say 'python -m scripts.script'.

Yeah. Start writing a program in 'myprogram.py' as things grow do the right thing and split a function out to its own file and import it. It doesn't work. Suddenly you need to learn a whole bunch about python modules and the import system and scripts vs modules, and some of the questions you have just literally have no good answer.
Post reply on HN