Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Conda: A package management disaster?

pyherald.com

Post reply on HN
Print view View original

111–120 of 234 posts

Re: Conda: A package management disaster?

#111

Post by theamk »

dxuh wrote: post #63
The_Colonel wrote: post #32

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the b…

I think at least part of it is that there are so many solutions for Python packaging, which are often intermixed or only half-supported by developers. It's a tough ask to provide dedicated support for pip, conda, poetry and what else is there plus a couple different ways to create virtual environments. Of course if you do everything right, you set it up once (if even that) and it just keeps working forever, but it is…

The problem is a lot of Python source is actually a C/C++ file, so simply having "source based package manager for Python" is very annoying, as you'd have to manage your C/C++ sources with some other mechanisms.

This is exactly the reason I've moved from pip to conda for some projects: "pip" was acting a source-based package manager, and thus asking for C tools, libraries and dev headers to be installed - but not providing them as they were non-Python and thus declared out of scope. Especially on older Linux distributions, getting dependencies right can be quite a task.

Re: Conda: A package management disaster?

#112

Post by theamk »

9dev wrote: post #93
cdavid wrote: post #55

Earlier quoted context omitted.

A lot of path dependency, but essentially 1. A good python solution needs to support native extensions. Few other languages solve this well, especially across unix + windows. 2. Python itself does not have package manager included. I am not sure solving 2 alone is enough, because it will be hard to fix 1 then. And ofc 2 would needs to have solution for older python versions. My guess is that we're stuck in a local ma…

PHP and composer do. You can specify native extensions in the composer.json file, along with an optional version requirement, and install them using composer just fine. Dependencies can in turn depend on specific extensions, or just recommend them without mandating an installation. This works across UNIX and Windows, as far as I’m aware.

does not seem so... Something as simple as "yaml" already requires reaching to apt-get: http://bd808.com/pecl-file_formats-yaml/

> Php-yaml can be installed using PHP's PECL package manager. This extension requires the LibYAML C library version 0.1.0 or higher to be installed.

    $ sudo apt-get install libyaml-dev
This is basically how "pip" works, and while it's fine for basic stuff, it gets pretty bad if you want to install fancy numerical of cryptography package on a LTS linux system that's at the end of the support period.

I am guessing that PHP might simply have less need for native packages, being more web-oriented.

Re: Conda: A package management disaster?

#113

Post by awestroke »

skeledrew wrote: post #102
TheAceOfHearts wrote: post #73

Earlier quoted context omitted.

The problem is that Python refuses to take responsibility for the whole ecosystem. One of the biggest success stories in programming language development has been Rust's realization that all of it matters: language, version management, package management, and build tools. To have a truly outstanding experience you need to take responsibility for the whole ecosystem. Python and many other older languages just focus on…

Who will pay for all this responsibility?

Yeah, who will pay for this drastic reduction in wasted time?

Re: Conda: A package management disaster?

#114

Post by duped »

throwawaymaths wrote: post #24
lmz wrote: post #22

Earlier quoted context omitted.

Wasn't node the only programming language that used a subdirectory for deps by default? Ruby and Perl certainly didn't have it - although Ruby did subsequently add Bundler to gems and gems supported multiversioning.

Rust, julia, elixir

Rust doesn't store dependencies under your project dir, but it does build them under your target.

Re: Conda: A package management disaster?

#115

Post by aaroninsf »

perrygeo wrote: post #104
The_Colonel wrote: post #32

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the b…

> what makes this so difficult to solve in Python? Python creates the perfect storm for package management hell: - Most the valuable libraries are natively compiled (so you get all the fun of distributing binaries for every platform without any of the traditional benefits of native compilation) - The dynamic nature makes it challenging to understand the non-local impacts of changes without a full integration test sui…

I agree with all of these and it makes me wonder as I do from time to time,

has anyone managed to make a viable P#, a clean break which retains most of what most people love about the language and environment; and cheerfully asserts new and immutable change in things like .

When I have looked into this it seems people can't help but improve one-more-thing or one-other-thing and end up just enjoying vaguely-pythonic language design.

Re: Conda: A package management disaster?

#116

Post by aaroninsf »

aaroninsf wrote: post #115
perrygeo wrote: post #104

Earlier quoted context omitted.

> what makes this so difficult to solve in Python? Python creates the perfect storm for package management hell: - Most the valuable libraries are natively compiled (so you get all the fun of distributing binaries for every platform without any of the traditional benefits of native compilation) - The dynamic nature makes it challenging to understand the non-local impacts of changes without a full integration test sui…

I agree with all of these and it makes me wonder as I do from time to time, has anyone managed to make a viable P#, a clean break which retains most of what most people love about the language and environment; and cheerfully asserts new and immutable change in things like . When I have looked into this it seems people can't help but improve one-more-thing or one-other-thing and end up just enjoying vaguely-pythonic l…

Googling P# led me to this delight which is 100% unrelated:

https://couragetotremble.blog/2007/08/09/p-language/

Re: Conda: A package management disaster?

#117

Post by graemep »

aaroninsf wrote: post #115
perrygeo wrote: post #104

Earlier quoted context omitted.

> what makes this so difficult to solve in Python? Python creates the perfect storm for package management hell: - Most the valuable libraries are natively compiled (so you get all the fun of distributing binaries for every platform without any of the traditional benefits of native compilation) - The dynamic nature makes it challenging to understand the non-local impacts of changes without a full integration test sui…

I agree with all of these and it makes me wonder as I do from time to time, has anyone managed to make a viable P#, a clean break which retains most of what most people love about the language and environment; and cheerfully asserts new and immutable change in things like . When I have looked into this it seems people can't help but improve one-more-thing or one-other-thing and end up just enjoying vaguely-pythonic l…

IronPython? The problem with that is compatibility with, and easy access to, existing libraries which is the main reason to use Python in the first place.

I also think some of the criticisms in the GP comment are not accurate. most of the valuable libraries are native compiled? Some important ones are, but not all.

I think a lot of the problem is that Python's usage has changed. Its great for a wide range of uses (scripting, web apps and other server stuff, even GUIs) but its really not a great match for scientific computing and the like but has become widely used there because it is easy to learn (and has lots of libraries for that now!).

Re: Conda: A package management disaster?

#118

Post by coliveira »

The_Colonel wrote: post #32

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the b…

It is not a new discovery that Python is terrible for packaging and distribution. Unfortunately, very little has been done about this. The fact that Python is used on particular environments controlled by the developers, mainly machine learning, makes this even more difficult to fix.

Re: Conda: A package management disaster?

#119

Post by cogman10 »

The_Colonel wrote: post #32

It's rare to see something as systematically broken as Python package/dependencies ecosystem. What I don't understand - what makes this so difficult to solve in Python? It seems that many other platforms solved this a long time ago - maven 2.0 was released almost 20 years ago. While it wasn't / isn't by no means perfect, its fundamentals were decent already back then. One thing which I think messed this up from the b…

> what makes this so difficult to solve in Python?

I think the answer is the same thing that makes it difficult to make a good package manager for C++.

When a language doesn't start with decent package management, it becomes really hard to retrofit a good one later in the lifespan of that language. Everyone can see "this sucks" but there's simply no good route to change the status quo.

I think Java is the one language I've seen that has successfully done the switch.

Re: Conda: A package management disaster?

#120

Post by michaelmrose »

wakawaka28 wrote: post #18
throwawaymaths wrote: post #12

Earlier quoted context omitted.

imagine being a beginner to programming and being told "use venvs" or worse, imagine being a longtime user of shells but not python and then being presented a venv as a solution to the problem that for some reason python doesn't stash deps in a subdirectory of your project

Beginners in Python typically don't need venvs. They can just install a few libraries (or no libraries even) to get started. If you truly need venvs then you're either past the initial learning phase or you're learning how to run Python apps instead of learning Python itself. For some libraries, it is not acceptable to stash the dependencies for every single toy app you use. I don't know how much space TensorFlow or…

Intelligent systems simply cache and re-use versions and do stash deps for every toy project without consuming space.

Also installing everything with pip is a great way to enjoy unexplainable breakage when a Doesn't work with v1 and b doesn't work with v2.

It also leads to breaking Linux systems where a large part of the system is python code. Especially where user upgrades system python for no reason.

Post reply on HN