How the Python import system works • Hacker News Board

Re: How the Python import system works

#21

Post by fullstop » Sat, Jul 24, 2021, 3:42 PM UTC

johncena33 wrote: post #3

I've been programming in Python professionally for more than five years. I consider myself quite a good programmer. I still don't have good grasp on Python's import system. Does anyone else have similar experience?

I sometimes fall into the trap of using pip to install dependencies and then things break after an os update. That is, my python version has changed from 3.8 to 3.9 and my dependencies are sitting in the wrong directory. I never know if I should use pip and requirements.txt or rely on Ubuntu's packaged versions.

Re: How the Python import system works

#22

Post by skindoe » Sat, Jul 24, 2021, 3:44 PM UTC

johncena33 wrote: post #3

I've been programming in Python professionally for more than five years. I consider myself quite a good programmer. I still don't have good grasp on Python's import system. Does anyone else have similar experience?

If you dig into importlib and try to write extensions for it then the underlying concepts of modules and packages will make a lot more sense.

For a project I was working on where you could dynamically call distributed tasks (we we're using ecs) I added a subclass of module and package that dynamically created package structures and modules from a database call.

So the new modules and packages would load from the database instead of from a python file and python the class would be dynamically generated to be something like albiet more advanced.

Class MyDistributedTask: Def run(self, input, execution_engine=ecs): Run task

So users of our package could import directly import their package_name.TaskName

And run it directly as long as they imported our package which contained a custom module loader to our db to discover.

Learned a ton about importing.

Re: How the Python import system works

#23

Post by jvolkman » Sat, Jul 24, 2021, 3:44 PM UTC

My least favorite is that import ordering matters in some situations. Like if I just run "organize imports", all of a sudden dependency cycles pop up and everything is broken. Certainly a sign of things being misimported/misorganized, but stuff happens has systems grow fast. And solving these issues is always incredibly time consuming.

Unless anyone knows of magical tools to help solve import issues?

Re: How the Python import system works

#24

Post by matheusmoreira » Sat, Jul 24, 2021, 3:46 PM UTC

It's certainly a lot better than Ruby's require which just executes code and alters global virtual machine state. Not too different from C's #include.

My favorite is Javascript's. Modules are objects containing functions and data, require returns such objects. Simple, elegant. Completely reifies the behind-the-scenes complexity of Python's import, making it easily understandable.

Re: How the Python import system works

#25

Post by brilee » Sat, Jul 24, 2021, 3:47 PM UTC

johncena33 wrote: post #3

I've been programming in Python professionally for more than five years. I consider myself quite a good programmer. I still don't have good grasp on Python's import system. Does anyone else have similar experience?

I'm a python readability approver at Google and I don't understand how the import system works

Re: How the Python import system works

#26

Post by BozeWolf » Sat, Jul 24, 2021, 3:47 PM UTC

abledon wrote: post #6

johncena33 wrote: post #3

I've been programming in Python professionally for more than five years. I consider myself quite a good programmer. I still don't have good grasp on Python's import system. Does anyone else have similar experience?

same! I've always been shielded from it with Django's conventions. (the ecosystem I mainly work in). I used a lot of '.' and '..' imports but I think something changed in python3 that made that strategy a lot less forgiving... now I _really_ should read the entirety of this article!

Always use absolute imports, do not use relative imports. Solves most problems. Also is recommended by pep8.

Skip the relative imports in the article. No need to read it entirely anymore ;-)

Re: How the Python import system works

#27

Post by sillysaurusx » Sat, Jul 24, 2021, 3:48 PM UTC

Fun fact: you can overload the Python import system to work with other languages that you create.

I use this for my Python-based Lisp: https://github.com/shawwn/pymen/blob/ml/importer.py

  import foo

checks for "foo.l", and if found, compiles it to foo.py on the fly and imports that instead.

It's so cursed. I love it.

Re: How the Python import system works

#28

Post by btilly » Sat, Jul 24, 2021, 3:50 PM UTC

maest wrote: post #13

Slightly off-topic: One of the simplest import systems I've seen were in q/kdb (like with most things in that language, everything is as simple as possible) Imports work by simply calling `\l my_script.q` which is similar to simply taking the file `my_script.q` and running it line by line (iirc, it does so eagerly, so it reruns the entire file whenever you do `\l my_script.q`, even if the file has been loaded before,…

Python avoids having to rerun the import over and over again.

Suppose that you are importing a larger project. Where your one import (say of your standard database interface) pulls in a group of modules (say one for each table in your database), all of which import a couple of base libraries (to make your database objects provide a common interface) that themselves pull in common Python modules (like psychopg2) which themselves do further imports of standard Python modules.

The combinatorial explosion of ways to get to those standard Python modules would make the redundant work of parsing them into a significant time commitment without a caching scheme.

From the point of view of a small script, this kind of design may seem like overkill. But in a large project, it is both reasonable and common.

Re: How the Python import system works

#29

Post by brundolf » Sat, Jul 24, 2021, 3:55 PM UTC

matheusmoreira wrote: post #24

It's certainly a lot better than Ruby's require which just executes code and alters global virtual machine state. Not too different from C's #include. My favorite is Javascript's. Modules are objects containing functions and data, require returns such objects. Simple, elegant. Completely reifies the behind-the-scenes complexity of Python's import, making it easily understandable.

Though that's not how JS's new module system works, which I would say is also elegant in its own ways, but juggling two different systems is less so

Re: How the Python import system works

#30

Post by abhisuri97 » Sat, Jul 24, 2021, 3:56 PM UTC

Despite using python for the past 4 years, it still takes me several tries to set up packages and imports correctly when I make them myself. Honestly, I wish that python had an import system similar to JS (where you can just say “I want this file” and specify the exports yourself). For me, it just feels more intuitive and less “magic”-like when dealing with custom scripts you want to import.