I really want to pay someone money to run package repo mirrors for me, but my problems have been more with npm than with Pypi. Astral, if you're listening.... maybe tackle JS packaging too?
PYX: The next step in Python packaging
111–120 of 479 posts
Re: PYX: The next step in Python packaging
#112Earlier quoted context omitted.
Stuff like that led me fully away from Ruby (due to Rails), which is a shame, I see videos of people chugging along with Ruby and loving it, and it looks like a fun language, but when the only way I can get a dev environment setup for Rails is using DigitalOcean droplets, I've lost all interest. It would always fail at compiling something for Rails. I would have loved to partake in the Rails hype back in 2012, but ov…
I'm surprised to hear that. Ruby was the first language in my life/career where I felt good about the dependency management and packaging solution. Even when I was a novice, I don't remember running into any problems that weren't obviously my fault (for example, installing the Ruby library for PostgreSQL before I had installed the Postgres libraries on the OS). Meanwhile, I didn't feel like Python had reached the bar…
Bundler has always been the best package manager of any language that I've used, but dealing with gem extensions can still be a pain. I've had lots of fun bugs where an extension worked in dev but not prod because of differences in library versions. I ended up creating a docker image for development that matched our production environment and that pretty much solved those problems.
Re: PYX: The next step in Python packaging
#113I'm brushing up with Python for a new job, and boy what a ride. Not because of the language itself but the tooling around packages. I'm coming from Go and TS/JS and while these two ecosystems have their own pros and cons, at least they are more or less straightforward to get onboarded (there are 1 or 2 tools you need to know about). In Python there are dozens of tools/concepts related to packaging: pip, easy_install,…
I appreciate everything they’ve done but the group which maintains Pip and the package index is categorically incapable of shipping anything at a good velocity. It’s entirely volunteer based so I don’t blame them, but the reality is that it’s holding back the ecosystem. I suspect it’s also a misalignment of interests. No one there really invests in improving UX.
> the group which maintains Pip and the package index is categorically incapable of shipping anything at a good velocity.
> It’s entirely volunteer based so I don’t blame them
It's not just that they're volunteers; it's the legacy codebase they're stuck with, and the use cases that people will expect them to continue supporting.
> I suspect it’s also a misalignment of interests. No one there really invests in improving UX.
"Invest" is the operative word here. When I read discussions in the community around tools like pip, a common theme is that the developers don't consider themselves competent to redesign the UX, and there is no money from anywhere to hire someone who would be. The PSF operates on an annual budget on the order of $4 million, and a big chunk of that is taken up by PyCon, supporting programs like PyLadies, generic marketing efforts, etc. Meanwhile, total bandwidth use at PyPI has crossed into the exabyte range (it was ~600 petabytes in 2023 and growing rapidly). They would be completely screwed without Fastly's incredible in-kind donation.
Re: PYX: The next step in Python packaging
#114Earlier quoted context omitted.
This is not a charitable interpretation. The more charitable read is that fixing search is non-trivial and has interlocking considerations that go beyond what pip's volunteer maintainers reasonably want to or can pick up. (And for the record: it isn't their fault at all . `pip search` doesn't work because PyPI removed the search API. PyPI removed that API for very good reasons[1].) [1]: https://github.com/pypa/pip/is…
That was 7 years ago. If it's not coming back, the CLI should make that clear, instead of giving a temporary "cannot connect" message that implies it could work, if you wait a minute and try again.
It was three years ago; 2018 is when they considered removing the command, not when the search API was actually removed from PyPI.
And this is part of the interlocking considerations I mentioned: there are private indices that supply the XML-RPC API, and breaking them doesn't seem justifiable[1].
Edit: fixed the link.
[1]: https://github.com/pypa/pip/issues/5216#issuecomment-1235329...
Re: PYX: The next step in Python packaging
#115Is this going to solve the combinatorial explosion of pre-building native dependencies for every possible target?
Python should get rid of its training wheels :^)
Re: PYX: The next step in Python packaging
#116>Modern I'll pass. I'd rather have the battle-tested old thing, thanks.
I've been limiting myself to whatever is available on debian and it's been fine for me since several years.
I don't understand why people who don't do weird AI stuff would use any of that instead of sticking to distribution packages and having the occasional 1 or 2 external modules that aren't packaged.
Re: PYX: The next step in Python packaging
#117Earlier quoted context omitted.
I appreciate everything they’ve done but the group which maintains Pip and the package index is categorically incapable of shipping anything at a good velocity. It’s entirely volunteer based so I don’t blame them, but the reality is that it’s holding back the ecosystem. I suspect it’s also a misalignment of interests. No one there really invests in improving UX.
Indeed, they broke a few features in the last few years and made the excuse "we can't support them, we're volunteers." Well, how about stop breaking things that worked for a decade? That would take less effort. They had time to force "--break-system-packages" on us though, something no one asked for.
> how about stop breaking things that worked for a decade?
They aren't doing this.
> They had time to force "--break-system-packages" on us though, something no one asked for.
The maintainers of several Linux distros asked for it very explicitly, and cooperated to design the feature. The rationale is extensively documented in the proposal (https://peps.python.org/pep-0668/). This is especially important for distros where the system package manager is itself implemented in Python, since corrupting the system Python environment could produce a state that is effectively unrecoverable (at least without detailed Python-specific know-how).
Re: PYX: The next step in Python packaging
#118I'm brushing up with Python for a new job, and boy what a ride. Not because of the language itself but the tooling around packages. I'm coming from Go and TS/JS and while these two ecosystems have their own pros and cons, at least they are more or less straightforward to get onboarded (there are 1 or 2 tools you need to know about). In Python there are dozens of tools/concepts related to packaging: pip, easy_install,…
> In Python there are dozens of tools/concepts related to packaging: pip, easy_install, setuptools, setup.py, pypy, poetry, uv, venv, virtualenv, pipenv, wheels, Some of those are package tools, some are dependency managers, some are runtime environments, some are package formats... Some are obsolete at this point, and others by necessity cover different portions of programming language technologies. I guess what I'm…
I don't know why you were downvoted. You are absolutely correct.
Re: PYX: The next step in Python packaging
#119> Why is it so hard to install PyTorch, or CUDA, or libraries like FlashAttention or DeepSpeed that build against PyTorch and CUDA? This is so true! On Windows (and WSL) it is also exacerbated by some packages requiring the use of compilers bundled with outdated Visual Studio versions, some of which are only available by manually crafting download paths. I can't wait for a better dev experience.
This was basically the reason to use anaconda back in the day.
Anaconda was a good idea until it would break apt on Ubuntu and make my job that much harder. That became the reason _not_ to use Anaconda in my book.
venv made these problems start to disappear, and now uv and Nix have closed the loop for me.
Re: PYX: The next step in Python packaging
#120Again! ezsetup, setuptools, conda, poetry, uv, now this.