Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PEP 810 – Explicit lazy imports

peps.python.org

Post reply on HN
Print view View original

111–120 of 247 posts

Re: PEP 810 – Explicit lazy imports

#111

Post by zahlman »

eesmith wrote: post #98
zahlman wrote: post #73

Earlier quoted context omitted.

With the LazyLoader technique I described at https://news.ycombinator.com/item?id=45467489 , there is no problem: >>> import nonexistent_module Traceback (most recent call last): File " ", line 1, in import nonexistent_module File " ", line 1360, in _find_and_load File " ", line 1322, in _find_and_load_unlocked File " ", line 1262, in _find_spec File " ", line 8, in find_spec base.loader = LazyLoader(base.loader) ^^^…

Ahh, so you do the find first, and keep that around before loading. I have bad memories of using a network filesystem where my Python app's startup time was 5 or more seconds because of all the small file lookups for the import were really slow. I fixed it by importing modules in functions, only when needed, so the time went down to less than a second. (It was even better using a zipimport, but for other reasons we d…

Yes, if checking for a file is slow, then checking for a file is slow. If you need to know up front whether the module exists, then you can't get around using the "figure out whether the module exists" machinery up front. And if the definition of "a module exists" includes cases where the module is represented by a file whose existence you have to check for, then there's no getting around that, either.

(Trying to do "fallback" logic with lazily-loaded modules is also susceptible to race conditions, of course. What if someone defines the module before you try to use it?)

Re: PEP 810 – Explicit lazy imports

#112

Post by zahlman »

aftbit wrote: post #94
zahlman wrote: post #68

Earlier quoted context omitted.

As a counterpoint, having all the imports automatically deferred would instantly dramatically speed up pip for short tasks. $ time pip install --disable-pip-version-check ERROR: You must give at least one requirement to install (see "pip help install") real 0m0.399s user 0m0.360s sys 0m0.041s Almost all of this time is spent importing (and later unloading) ultimately useless vendored code. From my testing (hacking th…

I think this makes a ton of sense in the very specific narrow use case of python CLI tools. For a web app or other long-lived process, startup time is typically not of extreme concern, and having more simplicity and legibility to the import process seems better. That's not to say this PEP should not be accepted. One could always apply a no-lazy-imports style rule or disable it via global lazy import control. https://…

It saddens me to think that the use case of "python CLI tools" is thought of as anything like "very specific" or "narrow".

Re: PEP 810 – Explicit lazy imports

#113

Post by zahlman »

hnlmorg wrote: post #109
zahlman wrote: post #103

Earlier quoted context omitted.

No, changing this breaks the world. A huge fraction of PyPI becomes completely invalid overnight, and the rest fails the expected version checks. Not to mention that the language is fundamentally designed around the expectation that modules are singleton. I've written about this at length before but I can't easily find it right now (I have way too many bookmarks and not nearly enough idea how to organize them). Yes,…

The hard part is making the change. Adding an escape hatch so older code still works is easy in comparison. Nobody is claiming this is a trivial problem to solve but its also not an impossible problem. Other languages have managed to figure out how to achieve this and still maintain backwards compatibility.

You're welcome to bring a concrete proposal to, e.g., https://discuss.python.org/c/ideas/6 , or ask around the core devs to find a PEP sponsor.

Note that you will be expected to have familiarized yourself generally with previous failed proposals of this sort, and proactively considered all the reasonably obvious corner cases.

Re: PEP 810 – Explicit lazy imports

#114

Post by thayne »

One thing the PEP doesn't really talk about, and that I find very annoying is that many python linters will complain if you don't put all of your imports at the top of the file, so you get lint warnings if you do the most obvious way to implement lazy imports.

And that is actually a problem for more than just performance. In some cases, importing at the top might actually just fail. For example if you need a platform specific library, but only if it is running on that platform.

Re: PEP 810 – Explicit lazy imports

#115

Post by charliermarsh »

veber-alex wrote: post #87
comex wrote: post #67

I don't hate it but I don't love it. It sounds like everyone will start writing `lazy` before essentially every single import, with rare exceptions where eager importing is actually needed. That makes Python code visually noisier. And with no plan to ever change the default, the noise will stay forever. I would have preferred a system where modules opt in to being lazy-loaded, with no extra syntax on the import side.…

I would gladly take a command line flag that I can pass to python that makes all module loading lazy. Unless you are writing scripts or very simple stuff running side effects when modules are loaded should be avoided at all cost anyway.

The PEP includes the ability to enable (or disable) lazy imports globally via a command-line flag or environment variable, in addition to the import syntax.

Re: PEP 810 – Explicit lazy imports

#116

Post by josefx »

f311a wrote: post #29
Alir3z4 wrote: post #25

I wish all imports were lazy by default. I know/heard there are "some" (which I haven't seen by the way) libraries that depend on import side effects, but the advantage is much bigger. First of all, the circular import problem will go away, especially on type hints. Although there was a PEP or recent addition to make the annotation not not cause such issue. Second and most important of all, is the launch time of Pyth…

Lazy imports mean late errors. Fail fast is a good design principle.

Python really seems like a bad fit for that. So your imports succeed, what now? Do they have all the functions or fields your program needs? Those are still resolved at the last possible moment. If you want to be sure your program actually runs you will have to write and run tests with and without lazy imports.

Re: PEP 810 – Explicit lazy imports

#117

Post by hnlmorg »

zahlman wrote: post #113
hnlmorg wrote: post #109

Earlier quoted context omitted.

The hard part is making the change. Adding an escape hatch so older code still works is easy in comparison. Nobody is claiming this is a trivial problem to solve but its also not an impossible problem. Other languages have managed to figure out how to achieve this and still maintain backwards compatibility.

You're welcome to bring a concrete proposal to, e.g., https://discuss.python.org/c/ideas/6 , or ask around the core devs to find a PEP sponsor. Note that you will be expected to have familiarized yourself generally with previous failed proposals of this sort, and proactively considered all the reasonably obvious corner cases.

Tbh you’re just reinforcing mine and others point there that the issue isn’t a technical one.

Re: PEP 810 – Explicit lazy imports

#119

Post by mekoka »

comex wrote: post #67

I don't hate it but I don't love it. It sounds like everyone will start writing `lazy` before essentially every single import, with rare exceptions where eager importing is actually needed. That makes Python code visually noisier. And with no plan to ever change the default, the noise will stay forever. I would have preferred a system where modules opt in to being lazy-loaded, with no extra syntax on the import side.…

If everyone starts favoring lazy imports with not much fuss then it means that lazy should have been the default behavior and eager is the keyword we're missing. This isn't the first time Python revisits this paradigm. Many constructs that used to eagerly produce lists in v2 were turned into generators in v3 with next to no problems.
Post reply on HN