71–80 of 401 posts
I've been mostly out of the python game for quite a while, but I never had that much issue with: pip install -r requirements.txt . Seems like a lot of people have tried their hand at various tooling, so there must be more to it than I am aware of.
This works if you only have one python project on your system, but most python developers need virtual environments to deal with various projects.
I've been mostly out of the python game for quite a while, but I never had that much issue with: pip install -r requirements.txt . Seems like a lot of people have tried their hand at various tooling, so there must be more to it than I am aware of.
Just two reasons (there are more): 1) uv is vastly faster than pip. Just using uv pip install -r requirements.txt and nothing else is a win. 2) uv can handle things like downloading the correct python person, creating a venv (or activating an existing venv if one exists) and essentially all the other cognitive load in a way that's completely transparent to the user. It means you can give someone a Python project and…
Earlier quoted context omitted.
Sounds like a great use case for ZFS’s deduplication at block level.
or, you know.. symlinks
Hardlink is somewhat better because both point to the same inode, but will also not work if the file needs different permissions or needs to be independently mutable from different locations.
Reflink hits the sweetspot where it can have different permissions, updates trigger CoW preventing confusing mutations, and all while still reducing total disk usage.
A very well written article! I admire the analysis done by the author regarding the difficulties of Python packaging. With the advent of uv, I'm finally feeling like Python packaging is solved. As mentioned in the article, being able to have inline dependencies in a single-file Python script and running it naturally is just beautiful. #!/usr/bin/env -S uv run # /// script # dependencies = ['requests', 'beautifulsoup4…
One other key part of this is freezing a timestamp with your dependency list, because Python packages are absolutely terrible at maintaining compatibility a year or three or five later as PyPI populates with newer and newer versions. The special toml incantation is [tool.uv] exclude-newer: # /// script # dependencies = [ # "requests", # ] # [tool.uv] # exclude-newer = "2023-10-16T00:00:00Z" # /// https://docs.astral.…
Can someone explain a non-project based workflow/configuration for uv? I get creating a bespoke folder, repo, and uv venv for certain long-lived projects (like creating different apps?). But most of my work, since I adopted conda 7ish years ago, involves using the same ML environment across any number of folders or even throw-away notebooks on the desktop, for instance. I’ll create the environment and sometimes add n…
`uv` isn't great for that, I've been specifying and rebuilding my environments for each "project". My one off notebook I'm going to set up to be similar to the scripts, will require some mods. It does take up a lot more space, it is quite a bit faster. However, you could use the workspace concept for this I believe, and have the dependencies for all the projects described in one root folder and then all sub-folders w…
FYI, for anyone else that stumbles upon this: I decided to do a quick check on PyTorch (the most problem-prone dependency I've had), and noticed that they recommending specifically no longer using conda—and have since last November.
Earlier quoted context omitted.
> Different Python versions each need their own separate Torch binaries as well Found this the hard way. Something to do with breakage in ABI perhaps. Was looking at the way python implements extensions the other day. Very weird.
> Something to do with breakage in ABI perhaps. Was looking at the way python implements extensions the other day. Very weird. Yes, it's essentially that: CPython doesn't guarantee exact ABI stability between versions unless the extension (and its enclosing package) intentionally build against the stable ABI[1]. The courteous thing to do in the Python packaging ecosystem is to build "abi3" wheels that are stable and…
Earlier quoted context omitted.
> Something to do with breakage in ABI perhaps. Was looking at the way python implements extensions the other day. Very weird. Yes, it's essentially that: CPython doesn't guarantee exact ABI stability between versions unless the extension (and its enclosing package) intentionally build against the stable ABI[1]. The courteous thing to do in the Python packaging ecosystem is to build "abi3" wheels that are stable and…
The reason why people don't always use abi3 is because not everything that can be done with the full API is even possible with the limited one, and some things that are possible carry a significant perf hit.
(I don't know if this is the reason in Torch's case or not, but I know from experience that it's the reason for many other popular Python packages.)
> I had a friend who decided to not use uv, because the first time he used it, it was on a 15 years old codebase that had just been migrated to Python 3. It was standing on a pile of never cleaned up pip freeze exports, and uv could not make it work. This is my only gripe with uv, despite how the author decided to depict it, this really turns into a headache fast as soon as you have ~4-5 in-house packages. I don't th…
What’s a typical or better way of handling in-house packages?
You don't have to love uv, and there are plenty of reasons not to.
I've been mostly out of the python game for quite a while, but I never had that much issue with: pip install -r requirements.txt . Seems like a lot of people have tried their hand at various tooling, so there must be more to it than I am aware of.
This works if you only have one python project on your system, but most python developers need virtual environments to deal with various projects.