Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PYX: The next step in Python packaging

astral.sh

Post reply on HN
Print view View original

441–450 of 479 posts

Re: PYX: The next step in Python packaging

#441

Post by notatallshaw »

imtringued wrote: post #364
notatallshaw wrote: post #92

Earlier quoted context omitted.

> However, pip has some really gnarly internal infrastructure that prevents it from taking advantage of a lot of uv's good ideas (which in turn are not all original). FWIW, as a pip maintainer, I don't strongly agree with this statement, I think if pip had the same full time employee resources that uv has enjoyed over the last year that a lot of these issues could be solved. I'm not saying here that pip doesn't have…

>FWIW, as a pip maintainer, I don't strongly agree with this statement, I think if pip had the same full time employee resources that uv has enjoyed over the last year that a lot of these issues could be solved. No. This something people tell each other, that it's a lack of resources, but in reality almost all OSS projects with long standing flaws don't have a resources problem. They have a prioritization problem, wh…

> No. This something people tell each other, that it's a lack of resources,

Well uv, as an OSS project, has come along with around 2 orders of magnitude more manpower than pip and has solved a lot of problems

> but in reality almost all OSS projects with long standing flaws don't have a resources problem. They have a prioritization problem, where they actively ignore and refuse to work on things that affect users every single day of usage.

I wasn't making a statement about OSS projects in general, but I agree that pip has prioritization problems but I would argue that it stems from the lack of resources.

The people who do donate their spare time are in fact only likely to donate that time on things that interest them. If there were resources to have someone act like a project manager, and other to follow their lead, then the prioritization problems could be fixed, but there aren't those resources.

> No more bullshit excuses

Then contribute your full time hours to pip, or other OSS that needs fixing, rather than arm chair commenting on hacker news.

Re: PYX: The next step in Python packaging

#442

Post by notatallshaw »

zahlman wrote: post #230
notatallshaw wrote: post #92

Earlier quoted context omitted.

> However, pip has some really gnarly internal infrastructure that prevents it from taking advantage of a lot of uv's good ideas (which in turn are not all original). FWIW, as a pip maintainer, I don't strongly agree with this statement, I think if pip had the same full time employee resources that uv has enjoyed over the last year that a lot of these issues could be solved. I'm not saying here that pip doesn't have…

Good to see you again. > FWIW, as a pip maintainer, I don't strongly agree with this statement, I think if pip had the same full time employee resources that uv has enjoyed over the last year that a lot of these issues could be solved. Fair enough. I'm sure if someone were paying me a competitive salary to develop my projects, they'd be getting done much faster, too. > I actually think this isn't a great example, evi…

> I guess you're talking about the fact that uv's cache only stores the unpacked version, rather than the original wheel? I'm planning to keep the wheel around, too. But my point was more that because of this cache structure, pip can't even just grab the wheel from its cache without hitting the Internet, on top of not having a place to put a cache of the unpacked files.

I'm talking about the fact there is no `uv pip download` or `uv pip wheel`.

I'm sure we'll discuss when you add this feature to uv, but I personally find uv's cache already grows too big too fast, so adding more to it makes me concerned.

> You mean, as opposed to obtaining it per version, lazily? Because it does seem like the .metadata file system works pretty well nowadays.

Yeah, one of the way uv speeds up resolving is it pre-downloads .metadata files and checks if their requirements are identical to versions it had already checked, so it can quickly rule them out. It's a clever use of a collector and an advanced resolver.

Re: PYX: The next step in Python packaging

#443

Post by aidenn0 »

zahlman wrote: post #152
mikepurvis wrote: post #83

Earlier quoted context omitted.

Another piece is that PyPI has no index— it's just a giant list of URLs [1] where any required metadata (eg, the OS, python version, etc) is encoded in the filename. That makes it trivial to throw behind a CDN since it's all super static, but it has some important limitations: - there's no way to do an installation dry run without pre-downloading all the packages (to get their dep info) - there's no way to get hashes…

Your information is out of date. > there's no way to do an installation dry run without pre-downloading all the packages (to get their dep info) Not true for wheels; PyPI implements https://peps.python.org/pep-0658/ here. You can pre-download just the dependency info instead. For sdists, this is impossible until we can drop support for a bunch of older packages that don't follow modern standards (which is to say, inc…

>> there's no way to get hashes of the archives

> This is provided as a URL fragment on the URLs, as described in https://peps.python.org/pep-0503/. Per PEP 658, the hash for the corresponding metadata files is provided in the data-dist-info-metadata (and data-core-metadata) attributes of the links.

Also, this is an implementation detail, but the archives are stored in a content-addressable store with the path being based on the (hexadecimal) blake2b/256 hash of their contents.

Re: PYX: The next step in Python packaging

#444

Post by notatallshaw »

quotemstr wrote: post #423
zahlman wrote: post #413

Earlier quoted context omitted.

> They have a prioritization problem This is what forks are for. Alternately, money would probably help. Otherwise it's a bit rude to impose your priorities on others. Perhaps some larger projects lose track of what they intended to prioritize. This is also a resources problem. There's nobody available to do management (and probably also not a social structure internally that would allow for it). > because of how "im…

imtringued is right though. Obstinate maintainership is a real failure mode of open source (and other) projects and it's often true that patch rejections involving phrases like "maintenance burden" are often pretextual. It's a shame to have to resort to creative destruction to improve things. Forks have a low success probability.

> Obstinate maintainership is a real failure mode of open source

I agree, but I also think sometimes that people think it's obstinate maintainership and it's not. For example, large PRs are hard to merge from a contributor who is fresh to the project because there is so much careful review that needs to be done, it's very resource intensive.

That said, one of my goals of becoming a maintainer has to be to try and making submitting a PR more friendly. Feel free to submit a PR to pip, I will be happy to help you get over any teething issues.

Re: PYX: The next step in Python packaging

#445

Post by bluenose69 »

orhmeh09 wrote: post #397
awesome_dude wrote: post #68

Earlier quoted context omitted.

Lets be honest here - whilst some experiences are better/worse than others, there doesn't seem to be a dependency management system that isn't (at least half) broken. I use Go a lot, the journey has been - No dependency management - Glide - Depmod - I forget the name of the precursor - I just remembered, VGo - Modules We still have proxying, vendoring, versioning problems Python: VirtualEnv Rust: Cargo Java: Maven an…

I think CRAN for R is very good, partly aided by an aggressive pruning policy for broken packages.

I agree. Here are some things that I (a science researcher and professor) like about R and CRAN:

1. There are a lot of build checks for problems involving mismatches between documentation and code, failed test suites, etc. These tests are run on the present R release, the last release, and the development version. And the tests are run on a routine basis. So, you can visit the CRAN site and tell at a glance whether the package has problems.

2. There is a convention in the community that code ought to be well-documented and well-tested. These tend not to be afterthoughts.)

3. if the author of package x makes changes, then all CRAN packages that use x will be tested (via the test suite) for new problems. This (again because of the convention of having good tests) prevents lots of ripple-effect problems.

4. Many CRAN packages come with so-called vignettes, which are essays that tend to supply a lot of useful information that does not quite fit into manpages for the functions in the package.

5. Many CRAN packages are paired with journal/textbook publications, which explain the methodologies, applications, limitations, etc in great detail.

6. CRAN has no problem rejecting packages, or removing packages that have problems that have gone unaddressed.

7. R resolves dependencies for the user and, since packages are pre-built for various machine/os types, installing packages is usually a quick operation.

PS. Julia is also very good on package management and testing. However, it lacks a central repository like CRAN and does not seem to have as strong a culture of pairing code with user-level documentation.

Re: PYX: The next step in Python packaging

#446

Post by woodruffw »

aragilar wrote: post #374
woodruffw wrote: post #277

Earlier quoted context omitted.

> I breathe a sigh of relief in language like Go and Rust with an “authoritative” built-in solution. I don't know about Go, but Rust's packaging isn't authoritative in the sense that I meant. There's no packaging BDFL; improvements to Rust packaging happen through a standards process that closely mirrors that of Python's PEPs. I think the actual difference between Rust and Python is that Rust made the (IMO correct) d…

Was not setuptools the single tool for package management? It provided both the installer plus hooks to define packages and manage the install (plus eggs to try to manage the path). That doesn't mean the choices were the right ones (the package definition being the only thing to survive), but it seems that while cargo has the mindshare for pure-rust, when there's the need for integration with other ecosystems (as is…

Kind of, except that setuptools was never part of the standard library or standard distribution of Python. So it suffers/suffered from the same bootstrapping issue as the rest of Python packaging.

Re: PYX: The next step in Python packaging

#447

Post by divbzero »

zanie wrote: post #174
jsmeaton wrote: post #171

Astral folks that are around - there seems to be a bit of confusion in the product page that the blog post makes a little more clear. > The next step in Python packaging The headline is the confusing bit I think - "oh no, another tool already?" IMO you should lean into stating this is going to be a paid product (answering how you plan to make money and become sustainable), and highlight that this will help solve priv…

Thanks for the feedback!

The combination of

– “client (uv) and server (pyx)” and

– “You can use it to host your own internal packages, or as an accelerated, configurable frontend to public sources like PyPI and the PyTorch index.”

is what really helped me understand what pyx aims to be.

Re: PYX: The next step in Python packaging

#448

Post by crystal_revenge »

monster_truck wrote: post #10

I've been burned too many times by embracing open source products like this. We've been fed promises like these before. They will inevitably get acquired. Years of documentation, issues, and pull requests will be deleted with little-to-no notice. An exclusively commercial replacement will materialize from the new company that is inexplicably missing the features you relied on in the first place.

I love using uv, but having worked for a VC funded open source startup, your concerns are spot on.

As soon as there is a commercial interest competing with the open source project at the same company the OSS version will begin to degrade, and often the OSS community will be left in the dark about this. The startup I was at had plenty of funding, far too many engineers, and still removed basically every internal resource from the oss project except one person and drove out everyone working on the community end of things.

I would also recommend avoiding working for any open source startup if your goal is to get paid to contribute to a community project. Plenty of devs will take a reduced salary to work on a great community project, but most of the engineers I saw definitely got the "bait and switch" and moved immediately to commercial projects.

Re: PYX: The next step in Python packaging

#449

Post by mbonnet »

hexo wrote: post #393
rmonvfer wrote: post #356

To be honest, this was just a matter of time. As a long time Python developer, I just can’t wrap my head around the lack of something like this. GitHub was going to get hosted packages for Python but never did because it “didn’t align with their strategy objectives and a reallocation of resources” [1] (or some other similar corpospeak) Astral is a great company and I think we can’t question what they’ve achieved and…

to be honest. ill never use uv. python ecosystem tools should be in python.

this is the kind of statement that will get anyone who works for me PIPd

Re: PYX: The next step in Python packaging

#450

Post by mbonnet »

runningmike wrote: post #103

All python packaging challenges are solved. Lesson learned is that there is not a single solution for all problems. getting more strings attached with VC funded companies and leaning on their infrastructure is a high risk for any FOSS community.

If Python packaging problems are solved, why is Python known for having the worst tooling ecosystem of any "modern" language?
Post reply on HN