Even better would be if you could specify the dependencies inside of the script.
Edit: might be possible now? https://simonwillison.net/2024/Dec/19/one-shot-python-tools/
Uv's killer feature is making ad-hoc environments easy
371–380 of 428 posts
Re: Uv's killer feature is making ad-hoc environments easy
#372Earlier quoted context omitted.
In principle, venvs could hard-link the files from a common source, as long as the filesystem supports that. I'm planning to experiment with this for Paper. It's also possible to use .pth files ( https://docs.python.org/3/library/site.html ) to add additional folders to the current environment at startup. (I've heard some whispers that this causes a performance hit, but I haven't noticed. Python module imports are ca…
I wish the ecosystem made heavier use of .zip packages, which would gravely help with the logistics of your hardlink plan, in addition to slimming down 300MB worth of source code. The bad news is that (AIUI) the code needs to be prepared for use from a package and thus my retroactively just zipping them up will break things at runtime Take for example: $ du -hs $HOMEBREW_PREFIX/Cellar/ansible/11.1.0_1/libexec/lib/pyt…
Wheels are zip files with a different extension. And the Python runtime can import Python code from them - that's crucial to the Pip bootstrapping process, in fact.
But packages that stay zipped aren't a thing any more - because it's the installer that gets to choose how to install the package, and Pip just doesn't do that. Pip unpacks the wheel mainly because many packages are written with the assumption that the code will be unpacked. For example, you can't `open` a relative path to a data file that you include with your code, if the code remains in a zip file. Back in the day when eggs were still a common distribution format, it used to be common to set metadata to say that it's safe to leave the package zipped up. But it turned out to be rather difficult to actually be sure that was true.
The wheel is also allowed to contain a folder with files that are supposed to go into other specified directories (specific ones described by the `sysconfig` standard library https://docs.python.org/3/library/sysconfig.html ), and Pip may also read some "entry point" metadata and create executable wrappers, and leave additional metadata to say things like "this package was installed with Pip". The full installation process for wheels as designed is described at https://packaging.python.org/en/latest/specifications/binary... . (Especially see the #is-it-possible-to-import-python-code-directly-from-a-wheel-file section at the end.)
The unzipped package doesn't just take up extra space due to unzipping, but also because of the .pyc cache files. A recent Pip wheel is 1.8 MiB packed and 15.2 MiB unpacked on my system; that's about 5.8 apparent MiB .py, 6.5 MiB .pyc, 2.1 MiB from wasted space in 4KiB disk blocks, and [s].8 MiB from a couple hundred folders.[/s] Sorry, most of that last bit is actually stub executables that Pip uses on Windows to make its wrappers "real" executables, because Windows treats .exe files specially.
Re: Uv's killer feature is making ad-hoc environments easy
#373Earlier quoted context omitted.
Look, I'm not trying to sell this as a full solution - I'm just trying to establish that a lot of people really don't need a full solution. >only works on the exact same Python version It works on any Python version that all of the dependencies work on. But also it can be worked around with environment markers, if you really can support multiple Python versions but need a different set of dependencies for each. In pr…
So many misunderstandings here :/ I can’t muster the energy to correct them past these two obvious ones > It works on any Python version that all of the dependencies work on No, it doesn’t. It’s not a lockfile: it’s a snapshot of the dependencies you have installed . The dependencies you have installed depend on the Python version and your OS. The obvious case would be requiring a Linux-only dependency on… Linux, or…
>So many misunderstandings here :/
I've spent most of the last two years making myself an expert on the topic of Python packaging. You can see this through the rest of the thread.
>No, it doesn’t. It’s not a lockfile: it’s a snapshot of the dependencies you have installed.
Yes, it does. It's a snapshot of the dependencies that you have installed. For each of those dependencies, there is some set of Python versions it supports. Collectively, the packages will work on the intersection of those sets of Python versions. (Because, for those Python versions, it will be possible to obtain working copies of each dependency at the specified version number.)
Which is what I said.
> The dependencies you have installed depend on the Python version and your OS. The obvious case would be requiring a Linux-only dependency on… Linux, or a package only required on Python A huge amount of packages are pure Python and work on a wide range of Python versions and have no OS dependency. In general, packages may have such restrictions, but do not necessarily. I know this because I've seen my own code working on a wide range of Python versions without making any particular effort to ensure that. It's policy for many popular packages to ensure they support all Python versions currently supported by the core Python dev team.
Looking beyond pure Python - if I depend on `numpy==2.2.1` (the most recent version at time of writing), that supports Python 3.10 through 3.13. As long as my other dependencies (and the code itself) don't impose further restrictions, the package will install on any of those Python versions. If you install my project on a different operating system, you may get a different wheel for version 2.2.1 of NumPy (the one that's appropriate for your system), but the code will still work. Because I tested it with version 2.2.1 of NumPy on my machine, and version 2.2.1 of Numpy on your machine (compiled for your machine) provides the same interface to my Python code, with the same semantics.
I'm not providing you with the wheel, so it doesn't matter that the wheel I install wouldn't work for you. I'm providing you(r copy of Pip) with the package name and version number; Pip takes care of the rest.
>You can continue to upload new wheels (or a sdist) long after a package version is initially released.
Sure, but that doesn't harm compatibility. In fact, I would be doing it specifically to improve compatibility. It wouldn't change what Pip chooses for your system, unless it's a better match for your system than previously available.
Re: Uv's killer feature is making ad-hoc environments easy
#374When using Maven build tool with Java, the downloaded artifacts always have the version number added to prefix (artifact-version.jar).. and that means there can be multiple versions stored in parallel, cached globally and cherry picked without any ambiguity. The first time when I used Node and Python, I was shocked that the version number is not part of any downloaded artifacts. Versioning the dependencies is such a fundamental need and having it part of the artifact file itself seems like a common sense to me. Can anyone please explain why the Python/Node build tools do not follow that?
Re: Uv's killer feature is making ad-hoc environments easy
#375Earlier quoted context omitted.
> pip might upgrade to a broken set of dependencies. I'm only aware of examples where it's the fault of the packages - i.e. they specify dependency version ranges that don't actually work for them (or stop working for them when a new version of the dependency is released). No tool can do anything about that on the user's end. > Or if you run on a different OS and the dependencies are different there (because of env m…
You can specify markers in the requirements file you write, not in the frozen requirements from 'pip freeze'. Because it's just a list of what's installed on your machine.
Running 'pip freeze' creates a plain text file. You can edit it to contain anything that would have been in "the requirements file you write". "Your requirements file" may or may not capture what it needs to, depending on how you created it. But Pip supports it. (And so does the `pyproject.toml` specification.)
Re: Uv's killer feature is making ad-hoc environments easy
#376Earlier quoted context omitted.
What actually, as an end user, about pnpm is better than Yarn? I've never found an advantage with pnpm in all the times I've tried it. They seem very 1:1 to me, but Yarn edges it out thanks to it having a plugin system and its ability to automatically pull `@types/` packages when needed.
automatically pull `@types/` packages when needed Wait, what? Since when?
It's a plugin that's included by default in Yarn 4: https://yarnpkg.com/api/plugin-typescript
In Yarn 2/3, it needs to be added manually.
Re: Uv's killer feature is making ad-hoc environments easy
#377Earlier quoted context omitted.
So many misunderstandings here :/ I can’t muster the energy to correct them past these two obvious ones > It works on any Python version that all of the dependencies work on No, it doesn’t. It’s not a lockfile: it’s a snapshot of the dependencies you have installed . The dependencies you have installed depend on the Python version and your OS. The obvious case would be requiring a Linux-only dependency on… Linux, or…
>So many misunderstandings here :/ I've spent most of the last two years making myself an expert on the topic of Python packaging. You can see this through the rest of the thread. >No, it doesn’t. It’s not a lockfile: it’s a snapshot of the dependencies you have installed. Yes, it does. It's a snapshot of the dependencies that you have installed. For each of those dependencies, there is some set of Python versions it…
Holy hell dude, you don’t need to write a novel for every reply. It’s not a lockfile because it’s a snapshot of what you have installed. End of.
It doesn’t handle environment markers nor is it reproducible. Given any non-trivial set of dependencies and/or more than 1 platform, it will lead to confusing issues.
Those confusing issues are the reason for lock files to exist, and the reason they are not just “the output of pip freeze”.
But you know this, given your two years of extensive expert study. Which I see very little evidence of.
Re: Uv's killer feature is making ad-hoc environments easy
#378Earlier quoted context omitted.
None of GP is about what Pip can technically do. It's about what a better tool still written in Python could do. The problems you're describing, or seeing solved with uv, don't seem to be about a problem with the design of virtual environments. (Uv still uses them.) They're about not having the paradigm of making a venv transiently , as part of the code invocation; or they're about not having a built-in automation of…
Why does it matter if it's written in python or not? I want the best tooling, don't care how it's made.
You are free to evaluate tooling by your own standards.
But it commonly comes across that people think it can't be written in Python if it's to have XYZ features, and by and large they're wrong, and I'm trying to point that out. In particular, people commonly seem to think that e.g. Pip needs to be in the same environment to work, and that's just not true. There's a system in place that defaults to copying Pip into every environment so that you can `python -m pip`, but this is wasteful and unnecessary. (Pip is designed to run under the install environment's Python, but this is a hacky implementation detail. It really just needs to know the destination paths and the target Python version.)
It also happens that I care about disk footprint quite a bit more than most people. Maybe because I still remember the computers I grew up with.
Re: Uv's killer feature is making ad-hoc environments easy
#379Earlier quoted context omitted.
>Nearly every other language solves this better than this. "Nearly every other language" determines the exact version of a library to use for you, when multiple versions would work, without you providing any input with which to make the decision? If you mean "I have had a more pleasant UX with the equivalent tasks in several other programming languages", that's justifiable and common, but not at all the same. >What y…
What's your process for ensuring all members of a large team are using the same versions of libraries in a non trivial python codebase?
If all you need to do is ensure everyone's on the same versions of the libraries - if you aren't concerned with your supply chain, and you can accept that members of your team are on different platforms and thus getting different wheels for the same version, and you don't have platform-specific dependency requirements - then pinned transitive dependencies are all the metadata you need. pyproject.toml isn't generally intended for this, unless what you're developing is purely an application that shouldn't ever be depended on by anyone else or sharing an environment with anything but its own dependencies. But it would work. The requirements.txt approach also works.
If you do have platform-specific dependency requirements, then you can't actually use the same versions of libraries, by definition. But you can e.g. specify those requirements abstractly, see what the installer produces on your platform, and produce a concrete requirement-set for others on platforms sufficiently similar to yours.
(I don't know offhand if any build backends out there will translate abstract dependencies from an sdist into concrete ones in a platform-specific wheel. Might be a nice feature for application devs.)
Of course there are people and organizations that have use cases for "real" lockfiles that list provenance and file hashes, and record metadata about the dependency graph, or whatever. But that's about more than just keeping a team in sync.
Re: Uv's killer feature is making ad-hoc environments easy
#380Earlier quoted context omitted.
>Nearly every other language solves this better than this. "Nearly every other language" determines the exact version of a library to use for you, when multiple versions would work, without you providing any input with which to make the decision? If you mean "I have had a more pleasant UX with the equivalent tasks in several other programming languages", that's justifiable and common, but not at all the same. >What y…
IMHO the clear separation between lockfile and deps in other package managers was a direct consequence of people being confused about what requirements.txt should be. It can be both and could be for ages (pip freeze) but the defaults were not conductive to clear separation. If we started with lockfile.txt and dependencies.txt, the world may have looked different. Alas.
The thing is, the distinction is purely semantic - Pip doesn't care. If you tell it all the exact versions of everything to install, it will still try to "solve" that - i.e., it will verify that what you've specified is mutually compatible, and check whether you left any dependencies out.