Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PEP 810 – Explicit lazy imports

peps.python.org

Post reply on HN
Print view View original

241–247 of 247 posts

Re: PEP 810 – Explicit lazy imports

#241

Post by zahlman »

lenkite wrote: post #237
zahlman wrote: post #209

Earlier quoted context omitted.

I don't follow your reasoning. `pyproject.toml` has nothing to do with what happens at runtime. It's about building and packaging the code. It also doesn't say anything related to the modules that will be imported at runtime. It deals in the names of distributions , which are completely independent of `import` statements.

What I mean is that the `lazy` keyword need not be specified in the source Python files. Instead you can mark off a bunch of modules identified by regex as lazy. Perhaps `pyproject.toml` isn't the best place and it should be an independent python file.

The issue is that Python code may validly stand apart from `pyproject.toml` or any other metadata.

Re: PEP 810 – Explicit lazy imports

#242

Post by zahlman »

driggs wrote: post #218
zahlman wrote: post #202

Earlier quoted context omitted.

> What do you do today to resolve a dependency conflict when an intermediate library has a just-so dependency tree? When an installer resolves dependency conflicts, the project code isn't running. The installer is free to discover new constraints on the fly, and to backtrack. It is in effect all being done "statically", in the sense of being ahead of the time that any other system cares about it being complete and co…

> and support having multiple simultaneous versions of any Python library installed. Installed. Not loaded. The reason is to do away with virtual environments. I just want to say `import numpy@2.3.x as np` in my code. If 2.3.2 is installed, it gets loaded as the singleton runtime library. If it's not installed, load the closest numpy available and print a warning to stderr. If a transient dependency in the runtime tr…

You'd have to invent a different way, within existing Python syntax, to communicate the version, but you can do this today with sys.path and sys.meta_path hacks.

But virtual environments are quite simply not a big deal. Installed libraries can be hard-linked and maybe even symlinked between environments and this can be set up very quickly. A virtual environment is defined by the pyvenv.cfg marker file; you don't need to use or even have activation scripts, and you especially don't (generally) need a separate copy of pip for each one, even if you do use pip.

On the flip side, allowing multiple versions of a library in a virtual environment has very little effect on package resolution; it just allows success in cases of conflict, but normally there aren't conflicts (because you're typically making a separate environment for a single "root" package, and it's supposed to be possible to use that package in Python as it actually exists, without hacks). The installer still has to scrounge up metadata (and discover it recursively) and check constraints.

Re: PEP 810 – Explicit lazy imports

#243

Post by zahlman »

redleader55 wrote: post #225
zahlman wrote: post #59

> The standard library provides the LazyLoader class to solve some of these inefficiency problems. It permits imports at the module level to work mostly like inline imports do. The use of these sorts of Python import internals is highly non-obvious. The Stack Overflow Q&A I found about it ( https://stackoverflow.com/questions/42703908/ ) doesn't result in an especially nice-looking UX. So here's a proof of concept in…

Can you explain why does "threading" needs to be loaded? The rest seems decently straightforward, but what initialization from threading is required and why only in 3.13+?

I didn't investigate deeply (I just happened to stumble on the issue while figuring out the implementation), but it might be needed in older versions too. The loading process instantiates a `threading.RLock`, but from a source code file that doesn't import that (because it's in a file that has to bootstrap the import process). I'm not entirely sure how `threading` is supposed to get imported normally. The other complicating factor is that I was testing this at the REPL, and 3.13 has a new REPL implementation which might change how those initial libraries are brought in.

Re: PEP 810 – Explicit lazy imports

#244

Post by zvr »

simonw wrote: post #32

Love this. My https://llm.datasette.io/ CLI tool supports plugins, and people were complaining about really slow start times even for commands like "llm --help" - it turned out there were popular plugins that did things like import pytorch at the base level, so the entire startup was blocked on heavy imports. I ended up adding a note to the plugin author docs suggesting lazy loading inside of functions - https://llm.…

It's not only in the case of plugins.

If a tool has different capabilities that use different imports, why load all of them if only a subset is required?

As a simple example, a tool that can generate output in various formats (e.g., json, csv, xml, ...) should only import the appropriate modules to handle the output format after having determined which ones will be used in this invocations.

Re: PEP 810 – Explicit lazy imports

#245

Post by aftbit »

zahlman wrote: post #112
aftbit wrote: post #94

Earlier quoted context omitted.

I think this makes a ton of sense in the very specific narrow use case of python CLI tools. For a web app or other long-lived process, startup time is typically not of extreme concern, and having more simplicity and legibility to the import process seems better. That's not to say this PEP should not be accepted. One could always apply a no-lazy-imports style rule or disable it via global lazy import control. https://…

It saddens me to think that the use case of "python CLI tools" is thought of as anything like "very specific" or "narrow".

There are tons of great CLI tools written in Python, but I would wager that the vast majority of Python code, no matter how you count it, isn't in a CLI tool. Certainly it's a worthy goal to speed them up especially given how frequently things like `pip` are run both by humans who have better things to do than wait on computers, and by automation which does things at giant scale. But most Python project and programmers who are not making CLI tools or other startup-latency sensitive stuff may be better served by the old default import style.

Re: PEP 810 – Explicit lazy imports

#246

Post by ndrezn »

flare_blitz wrote: post #135
charliermarsh wrote: post #21

Lazy imports have been proposed before, and were rejected most recently back in 2022: https://discuss.python.org/t/pep-690-lazy-imports-again/1966... . If I recall correctly, lazy imports are a feature supported in Cinder, Meta's version of CPython, and the PEP was driven by folks that worked on Cinder. Last time, a lot of the discussion centered around questions like: Should this be opt-in or opt-out? At what level?…

I also hope this proposal succeeds, but I'm not optimistic. This will break tons of code and introduce a slew of footguns. Import statements fundamentally have side effects, and when and how these side effects are applied will cause mysterious breakages that will keep people up for many nights. This is not fearmongering. There is a reason why the only flavor of Python with lazy imports comes from Meta, which is one o…

Product manager for Dash here At Plotly we're actually pretty excited about the potential for lazy loaded imports, as it could help out a lot with the import performance of Plotly.py.

As this comment mentions Dash apps would not support lazy loaded imports until the underlying Dash library changes how it loads in callbacks and component libraries (the two features which would be most impacted here), but that doesn't mean there's no path to success. We've been discussing some ways we could resolve this internally and if this PEP is accepted we'd certainly go further to see if we can fully support lazy loaded imports (of both the Dash library itself/Dash component libraries and for relative imports in Dash apps).

Re: PEP 810 – Explicit lazy imports

#247

Post by zahlman »

aftbit wrote: post #245
zahlman wrote: post #112

Earlier quoted context omitted.

It saddens me to think that the use case of "python CLI tools" is thought of as anything like "very specific" or "narrow".

There are tons of great CLI tools written in Python, but I would wager that the vast majority of Python code, no matter how you count it, isn't in a CLI tool. Certainly it's a worthy goal to speed them up especially given how frequently things like `pip` are run both by humans who have better things to do than wait on computers, and by automation which does things at giant scale. But most Python project and programme…

> But most Python project and programmers who are not making CLI tools or other startup-latency sensitive stuff may be better served by the old default import style.

That may well be the case, and Python's development processes would definitely tend to assume the same. That has a lot to do with why PEP 690 was rejected, and why this proposal is opt-in even though many people are concerned that a lot of projects will "opt in everywhere" and create a lot of noise.

Believe it or not, the process is actually very conservative. People complaining about the "churn" caused by deprecations and removals seem not to have any concept of how few suggestions actually get implemented, and how many are rejected (including ones that perennially occur to many new users). A browse through "ideas" forum where new pre-PEP ideas are commonly pitched (https://discuss.python.org/c/ideas/6) gives one the impression of a leisurely stroll through a graveyard.

And many people (including myself) can tell you that you'll often be put through a run-around: if your idea is good and can be implemented, then surely it falls on you to make and publicize (!) a third-party package, to prove the demand and the community support for your specific implementation; but if you somehow get there, now it's trivial for people to install support, so it doesn't need to be in the standard library (cf. Requests).

Post reply on HN