Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PEP 810 – Explicit lazy imports

peps.python.org

Post reply on HN
Print view View original

91–100 of 247 posts

Re: PEP 810 – Explicit lazy imports

#91

Post by zahlman »

f311a wrote: post #29
Alir3z4 wrote: post #25

I wish all imports were lazy by default. I know/heard there are "some" (which I haven't seen by the way) libraries that depend on import side effects, but the advantage is much bigger. First of all, the circular import problem will go away, especially on type hints. Although there was a PEP or recent addition to make the annotation not not cause such issue. Second and most important of all, is the launch time of Pyth…

Lazy imports mean late errors. Fail fast is a good design principle.

Top-level code should not be able to fail except in incredibly deterministic ways that are tested during development. Failing fast is not as good as not failing at all. Lazy imports mean the power to avoid importing things that don't need to be imported at all on this run. Good design also cares about performance to some extent. On my machine, asking pip to do literally nothing takes several times as long as creating a new virtual environment --without-pip .

Re: PEP 810 – Explicit lazy imports

#92

Post by aftbit »

maest wrote: post #34
aftbit wrote: post #33

We tend to prefer explicit top-level imports specifically because they reveal dependency problems as soon as the program starts, rather than potentially hours or days later when a specific code path is executed.

Who is "we"?

My current team at my current company (see bio if you're really interested), though I should say I'm not authorized to speak on behalf of my employer, so I should really say something more like "I".

Re: PEP 810 – Explicit lazy imports

#93

Post by stavros »

peterfirefly wrote: post #83
simonw wrote: post #32

Love this. My https://llm.datasette.io/ CLI tool supports plugins, and people were complaining about really slow start times even for commands like "llm --help" - it turned out there were popular plugins that did things like import pytorch at the base level, so the entire startup was blocked on heavy imports. I ended up adding a note to the plugin author docs suggesting lazy loading inside of functions - https://llm.…

Parse the command line and do things like "--help" without doing the imports. Only do imports when you know you need them -- or as an easy approximation, only if the easy command line options have been handled and there's still something to do.

Well yes, or you can just use the `lazy` keyword, when it makes it into core.

Re: PEP 810 – Explicit lazy imports

#94

Post by aftbit »

zahlman wrote: post #68
aftbit wrote: post #33

We tend to prefer explicit top-level imports specifically because they reveal dependency problems as soon as the program starts, rather than potentially hours or days later when a specific code path is executed.

As a counterpoint, having all the imports automatically deferred would instantly dramatically speed up pip for short tasks. $ time pip install --disable-pip-version-check ERROR: You must give at least one requirement to install (see "pip help install") real 0m0.399s user 0m0.360s sys 0m0.041s Almost all of this time is spent importing (and later unloading) ultimately useless vendored code. From my testing (hacking th…

I think this makes a ton of sense in the very specific narrow use case of python CLI tools. For a web app or other long-lived process, startup time is typically not of extreme concern, and having more simplicity and legibility to the import process seems better.

That's not to say this PEP should not be accepted. One could always apply a no-lazy-imports style rule or disable it via global lazy import control.

https://peps.python.org/pep-0810/#global-lazy-imports-contro...

Re: PEP 810 – Explicit lazy imports

#95

Post by colpabar »

matsemann wrote: post #24
colpabar wrote: post #14

Earlier quoted context omitted.

This is what I thought of too. I really only know python, do other languages not have that issue? In python it does not seem like a "problem" to me - whenever I have seen circular import issues it is because the code is organized poorly. I worry that this feature will lead to devs "fixing" circular import issues by using lazy imports.

Sometimes it's hard to avoid cyclic imports, without blaming the design. Like if a Parent has a Child, and the Child needs to know of the parent. Only way to solve that in python is to put everything in the same file, which also feels like bad deisgn.

I would say in that case, the Parent and Child shouldn't need to know about each other - some kind of handler in a different file should.

Although I guess that doesn't work in all cases, like defining foreign key relationships when using an orm (like sqlalchemy) for example. But in the orm case, the way to get around that is... lazy resolution :^)

Re: PEP 810 – Explicit lazy imports

#96

Post by IshKebab »

zahlman wrote: post #88
IshKebab wrote: post #8

Wake me up when we can import a module by relative file path.

Relative imports have been supported for approximately forever ( https://stackoverflow.com/questions/72852 ). If you mean "by explicitly specifying a path string" (as opposed to a symbolic name), that has also been supported for approximately forever ( https://stackoverflow.com/questions/67631 ). Today, the `importlib` standard library exposes all the steps of the import process — including figuring out where the sou…

Nope. Relative imports work by relative package path, which is not at all the same. Often when you run Python you don't even have a package path.

Using `importlib` is a horrible hack that breaks basically all tooling. You very very obviously are not supposed to do that.

Re: PEP 810 – Explicit lazy imports

#97

Post by xdfgh1112 »

veber-alex wrote: post #87
comex wrote: post #67

I don't hate it but I don't love it. It sounds like everyone will start writing `lazy` before essentially every single import, with rare exceptions where eager importing is actually needed. That makes Python code visually noisier. And with no plan to ever change the default, the noise will stay forever. I would have preferred a system where modules opt in to being lazy-loaded, with no extra syntax on the import side.…

I would gladly take a command line flag that I can pass to python that makes all module loading lazy. Unless you are writing scripts or very simple stuff running side effects when modules are loaded should be avoided at all cost anyway.

That's already part of the PIP. There is a flag to enable lazy imports for all possible imports.

Re: PEP 810 – Explicit lazy imports

#98

Post by eesmith »

zahlman wrote: post #73
eesmith wrote: post #17

Earlier quoted context omitted.

All my code which uses import probing would fail, such as fallbacks: try: import module except ImportError: import slow_module as module Conditional support testing would also break, like having tests which only run if module2 is available: try: import module2 except ImportError: def if_has_module2(f): return unittest.skip("module2 not available")(f) else: def if_has_module2(f): return f @if_has_module2 class TestMod…

With the LazyLoader technique I described at https://news.ycombinator.com/item?id=45467489 , there is no problem: >>> import nonexistent_module Traceback (most recent call last): File " ", line 1, in import nonexistent_module File " ", line 1360, in _find_and_load File " ", line 1322, in _find_and_load_unlocked File " ", line 1262, in _find_spec File " ", line 8, in find_spec base.loader = LazyLoader(base.loader) ^^^…

Ahh, so you do the find first, and keep that around before loading.

I have bad memories of using a network filesystem where my Python app's startup time was 5 or more seconds because of all the small file lookups for the import were really slow.

I fixed it by importing modules in functions, only when needed, so the time went down to less than a second. (It was even better using a zipimport, but for other reasons we didn't use that option.)

If I understand things correctly, your code would have the same several-second delay as it tries to resolve everything?

Re: PEP 810 – Explicit lazy imports

#99

Post by enriquto »

what is the point of this? you can just import inside function definitions:

    def antislash(A, b):
        from numpy.linalg import solve
        return solve(A, b)
thus numpy.linalg is only imported the first time you call the antislash function. Much cleaner than a global import.

Ignore wrong traditions. Put all imports in the innermost scopes of your code!

Re: PEP 810 – Explicit lazy imports

#100

Post by hnlmorg »

BiteCode_dev wrote: post #72
comex wrote: post #67

I don't hate it but I don't love it. It sounds like everyone will start writing `lazy` before essentially every single import, with rare exceptions where eager importing is actually needed. That makes Python code visually noisier. And with no plan to ever change the default, the noise will stay forever. I would have preferred a system where modules opt in to being lazy-loaded, with no extra syntax on the import side.…

We heard that about types, the walrus, asyncio, dataclasses and so much more. But it didn't happen, if people don't need something (and many don't know it exists or what it does), it's unlikely they use it. In fact, half of the community basically uses only a modernized set of python 2.4 features and that's one of the beauties of the language. You don't need a lot to be productive, and if you want more, you can optio…

People said the same about Perl and its “there’s more than one way to do things” ethos, which gained much criticism.

Same is true for C++.

In this specific case, I think a lazy load directive isn’t a bad addition. But one does need to be careful about adding new language features just because you have an active community.

Post reply on HN