Constraints Are Good: Python's Metadata Dilemma
lucumr.pocoo.org
Constraints Are Good: Python's Metadata Dilemma
1–10 of 26 posts
Re: Constraints Are Good: Python's Metadata Dilemma
#2Eggs are dying out, pointed out by this 2 year old blog post:
https://about.scarf.sh/post/python-wheels-vs-eggs
The metadata problem is related to the problem that pip had an unsound resolution algorithm based on "try to resolve something optimistically and hope it works when you get stuck and try to backtrack".
I did a lot of research along the line that led to uv 5 years ago and came to the conclusion that installing out of wheels you can set up a SMT problem the same way maven does and solve it right the first time. They had a PEP to publish metadata files for wheels in PyPi but I'd built something before that could suck the metadata out of a wheel with just 3 http range requests. I believed that any given project might depend on a legacy egg and in those cases you can build that egg into a wheel via a special process and store it in a private repo (a must for the perfect Python build system)
Re: Constraints Are Good: Python's Metadata Dilemma
#3I am curious how Python got into this situation. Was it largely taking the path of least resistance to more and more adoption?
I get that Python is, strictly speaking, an older language. But, it isn't like these are at all new considerations.
Re: Constraints Are Good: Python's Metadata Dilemma
#4> The challenge with dynamic metadata in Python is vast, but unless you are writing a resolver or packaging tool, you're not going to experience the pain as much.
But that is by choice, I as a user, am forced to debug this pile of garbage whenever things go wrong, so in a way it's even worse for users. It's a running joke in the machine learning community that the hard part about machine learning is having to deal with python packages.
Re: Constraints Are Good: Python's Metadata Dilemma
#5I am curious how Python got into this situation. Was it largely taking the path of least resistance to more and more adoption? I get that Python is, strictly speaking, an older language. But, it isn't like these are at all new considerations.
Classic case of lack of constraints early on. Once people use all that power you end up with a mess.
Re: Constraints Are Good: Python's Metadata Dilemma
#6Eggs are dying out, pointed out by this 2 year old blog post: https://about.scarf.sh/post/python-wheels-vs-eggs The metadata problem is related to the problem that pip had an unsound resolution algorithm based on "try to resolve something optimistically and hope it works when you get stuck and try to backtrack". I did a lot of research along the line that led to uv 5 years ago and came to the conclusion that installi…
The metadata problem is unrelated to eggs. Eggs haven’t played much of a role in a long time but the metadata system still exists.
Range requests are used by both uv and pip if the index supports it, but they have to make educated guesses about how reliable that metadata is.
The main problem are local packages during development and source distributions.
Re: Constraints Are Good: Python's Metadata Dilemma
#7Eggs are dying out, pointed out by this 2 year old blog post: https://about.scarf.sh/post/python-wheels-vs-eggs The metadata problem is related to the problem that pip had an unsound resolution algorithm based on "try to resolve something optimistically and hope it works when you get stuck and try to backtrack". I did a lot of research along the line that led to uv 5 years ago and came to the conclusion that installi…
The metadata problem is unrelated to eggs. Eggs haven’t played much of a role in a long time but the metadata system still exists. Range requests are used by both uv and pip if the index supports it, but they have to make educated guesses about how reliable that metadata is. The main problem are local packages during development and source distributions.
Back in the case of eggs you couldn't count on having the metadata until you ran setup.py which forced pip to be unreliable because so much stuff got installed and uninstalled in the process of a build.
There is a need for a complete answer for dev and private builds, I'll grant that. Private repos like we are used to in maven would help.
Re: Constraints Are Good: Python's Metadata Dilemma
#8Earlier quoted context omitted.
The metadata problem is unrelated to eggs. Eggs haven’t played much of a role in a long time but the metadata system still exists. Range requests are used by both uv and pip if the index supports it, but they have to make educated guesses about how reliable that metadata is. The main problem are local packages during development and source distributions.
Back in the case of eggs you couldn't count on having the metadata until you ran setup.py which forced pip to be unreliable because so much stuff got installed and uninstalled in the process of a build. There is a need for a complete answer for dev and private builds, I'll grant that. Private repos like we are used to in maven would help.
Eggs did not contain setup.py files. The metadata like for wheels was embedded in the egg (in the EGG-INFO folder) from my recollection. Eggs were zip importable after all.
Re: Constraints Are Good: Python's Metadata Dilemma
#9A lot of the problem seems to be driven by a desire to have editable installs. I personally have never understood why having editable installs is such an important need. When I'm working on a Python package and need to test something, I just run
python -m pip install --user
and I now have a local installation that I can use for testing.
Re: Constraints Are Good: Python's Metadata Dilemma
#10A lot of the problem seems to be driven by a desire to have editable installs. I personally have never understood why having editable installs is such an important need. When I'm working on a Python package and need to test something, I just run python -m pip install --user and I now have a local installation that I can use for testing.
That would you require to make re-installations if your local app you develop against after every code change. Very few people will want to do that and it’s potentially very slow.
It’s also a step not needed by most other ecosystems.