Pipenv: promises a lot, delivers very little • Hacker News Board

Re: Pipenv: promises a lot, delivers very little

#161

Post by strzibny » Thu, Dec 06, 2018, 9:18 AM UTC

erikb wrote: post #37

sigmonsays wrote: post #3

This article is just a reminder that it may be impossible to fix the nightmare that is pythons packaging system. I'll save the python rant but this kind of software just further fragments and damages pythons reputation.

Honestly it's a nightmare in every language. And I found in comparison to most other languages I've worked with (Java, Ruby, PHP, Golang) I actually feel it's done quite well. It's a super complex topic, it needs leadership support so that you can get a reasonable percentage of the community to drill down on one solution instead of having 4-5 competing ones, but it also needs a lot of in-depth insights, which is kind…

I always felt like RubyGems is mostly done right. And not just packaging itself, it's also easy to host your own gems and combine sources with upstream for example. What is that pipenv made better than RubyGems for you?

Re: Pipenv: promises a lot, delivers very little

#162

Post by ProblemFactory » Thu, Dec 06, 2018, 9:22 AM UTC

icebraining wrote: post #71

wirrbel wrote: post #62

Earlier quoted context omitted.

Basically it seems you are lacking only two shell commands. virtualenv venv . venv/bin/activate Then everything works with pip pip install numpy I actually prefer the pipenv way (or the virtualenvwrapper way) of putting virtualenvs into a dedicated location, so that I can wipe them off the hard disk if I want to free some space. The real interesting and occasionally bothering point on managing your virtualenv is reso…

At a previous job, we used an even simpler solution: just download everything to a directory (called "deps") with pip install --target deps -r requirements.txt Then running the application with PYTHONPATH=deps python myapp.py This was also simple to integrate with service supervisors, since most make it easy to configure a environment variable.

Virtualenv tip: you can skip the Bash magic of calling "activate" by providing the path to the python executable.

    /hello/venvs/myproject/bin/python myscript.py

will run myscript.py exactly as if the virtualenv was active, which is easier to configure for cron jobs and services.

Re: Pipenv: promises a lot, delivers very little

#163

Post by ProblemFactory » Thu, Dec 06, 2018, 9:39 AM UTC

wuliwong wrote: post #72

greysteil wrote: post #29

Earlier quoted context omitted.

FWIW, resolution in Python is significantly harder than for Ruby, JS or Elixir because it requires cloning down each dependency. To really speed it up, what's needed is a registry API that provides all of the details for resolution. That's much harder for Python because of the legacy of setup.py, which allows version resolution to depend on arbitrary system considerations.

I've never really delved deeply into package management. I've been using ruby for a while and using rbenv has made it pretty painless for a while. I've been using pipenv on my local machine but then switched to virtualenv in my ec2 deployments (because all the tutorials used it). What makes it easier for ruby? Is there just this "registry API" that has gained enough traction that everyone uses it?

The fundamental problem with Python dependencies is that they are calculated while executing setup.py, not declared statically.

For example, the popular scikit-learn package has the following in its setup.py:

    if platform.python_implementation() == 'PyPy':
        SCIPY_MIN_VERSION = '1.1.0'
        NUMPY_MIN_VERSION = '1.14.0'
    else:
        SCIPY_MIN_VERSION = '0.13.3'
        NUMPY_MIN_VERSION = '1.8.2'

and these are used to request dependencies from the resolver/downloader. You could have more dynamic dependencies that vary based on system libraries and tools installed on the machine where you are running it, or even the current weather.

Until this is replaced by static version numbers, and all popular packages adopt it, a registry API cannot exist as it needs to run code on your machine to figure out the dependencies.

Re: Pipenv: promises a lot, delivers very little

#164

Post by erfgh » Thu, Dec 06, 2018, 9:39 AM UTC

umvi wrote: post #54

I hate python package/dependency/virtualenv management so much. Even JavaScript is preferable. I find npm very easy to wrap my head around. I do npm install ___ and it does a lookup in its repository and downloads it and its dependencies to node_modules. If I want to start fresh, I simply delete node_modules. Everything else "just works" when I invoke node myapp.js. Punto e basta. pipenv masquarades as the same thing…

In python you just create a virtualenv with "virtualenv whatever". From then on everything is installed in there. If you want to start fresh, you just delete the virtualenv. How is this difficult?

Re: Pipenv: promises a lot, delivers very little

#165

Post by Normal_gaussian » Thu, Dec 06, 2018, 9:41 AM UTC

Evidlo wrote: post #155

umvi wrote: post #54

I hate python package/dependency/virtualenv management so much. Even JavaScript is preferable. I find npm very easy to wrap my head around. I do npm install ___ and it does a lookup in its repository and downloads it and its dependencies to node_modules. If I want to start fresh, I simply delete node_modules. Everything else "just works" when I invoke node myapp.js. Punto e basta. pipenv masquarades as the same thing…

I don't get the node way of doing things. Why should the availability of modules be dependent on the current working directory? Seems brittle to me that your app can't find its libraries because you ran it one directory up/down from where it should be. As for sudo, there is no situation in which you should use sudo with pip.

Node resolves relative to the path of the file requiring the module, not relative to the process cwd.

Re: Pipenv: promises a lot, delivers very little

#166

Post by gjvc » Thu, Dec 06, 2018, 9:45 AM UTC

What does pipenv deliver that cannot be done with pip and virtualenv?

Re: Pipenv: promises a lot, delivers very little

#167

Post by beagle3 » Thu, Dec 06, 2018, 9:47 AM UTC

dfee wrote: post #2

I was sorely disappointed with pipenv, and transitioned to poetry [1], with which I’ve been very satisfied. There is also some commentary in the README on the design decisions re: pipenv [3]. Contrary to the author's perspective on poetry using poetry-specific sections of pyproject.toml, that's actually the proper implementation (and expected usage) coming out of PEP-518. I also am a big fan of pyenv [3] but that’s o…

As you have experience with pyenv, pipenv and poetry, you might be able to answer: Why not conda?

I've been using conda since ~4 years now, and every single complaint lodged against any of the other package managers was never an issue with conda in the first place. And yet, it seems like there's a SEP field around it and people just ignore its existence?

In this thread, for the first time, I've seen someone mention that you might have problem porting a conda env from a Mac to Linux - never had that problem myself, but I guess it's possible; But that's easily solvable, and certainly does not require a new package manager?

Re: Pipenv: promises a lot, delivers very little

#168

Post by gvalkov » Thu, Dec 06, 2018, 9:52 AM UTC

sametmax wrote: post #68

legostormtroopr wrote: post #4

Pipenv is a really interesting development for Python, and I'm glad that someone was working to improve dependency locking for Python. However, Kenneth abused his position with PyPA (and quickly bumped a what is a beta product to version 18) to imply Pipenv was more stable, more supported and more official than it really was. And worse still, for anyone saying "but ts open source, you get what you pay for", Kenneth a…

I'm very glad we have the wheel and ensurepip now. Yet, I think PyPA has not been taking the best decisions regarding Python packaging. Your Kenneth story is not the only "weird event" in their history. E.G: Did you know that we don't need "pyproject.toml" at all ? That there is already a production ready plain text standard to replace setup.py ? Did you know that this standard has been perfectly working for TWO YEAR…

  > Use it. Talk about it. Write about it.

I have a project that converts basic setup.py files to setup.cfg files [1].

Still happily using plain setuptools for library development and pip-tools for application development.

[1]: https://github.com/gvalkov/setuptools-py2cfg

Re: Pipenv: promises a lot, delivers very little

#169

Post by jabl » Thu, Dec 06, 2018, 9:54 AM UTC

RayDonnelly wrote: post #119

epage wrote: post #38

Earlier quoted context omitted.

Poetry is one of the few things that gives me hope about the mess of python packaging. It is also great to see the author is very responsive. My only concern is the lack of integrated "toolchain" management (what versxon of python to use, something like rustup) that is cross platform.

I agree that providing toolchains is very important. The only non-system package manager that provides Python and its own toolchains - for Linux and macOS presently - which are used to compile every C, C++ and Fortran package, including Python itself is conda and the Anaconda Distribution. Not doing this leads to static linking and that's inefficient and insecure. Disclaimer: I work for Anaconda Inc.

conda is decent, but for some reason it seems to live in a weird parallel universe where people not in science have never heard about it.

While I'm generally happy with it, some gripes:

- Using it's own package format with its own repos means that for many (most) projects you can't get all dependencies from conda, but some from pypi as well.

- And it doesn't keep track of which files belong to which package. So package X will happily scribble over files installed by package Y, and vice versa, leading to either X or Y being silently broken depending on the order they were installed in! Argh! I mean, this is something dpkg/rpm/etc. figured out decades ago, it's not rocket science.

- The dependency solver seems a bit weird. Often when upgrading an environment, it will install the same version of a package with another 'build tag', then a few days later if you upgrade again, it will downgrade back to the previous build tag. Not sure if this is the fault of the dependency solver, or whether the problem is in the packages themselves.

- Similarly, there's a lot of mutual incompatibility in the repos. E.g. dependencies on openssl versions prevent upgrading, or require removal of some package etc. I think this is not so much the fault of the conda tool itself, but rather that Anaconda Inc. needs to be more picky wrt packaging policy. Again, Linux distros have been pretty good at this. E.g. https://www.debian.org/doc/debian-policy/ , https://docs.fedoraproject.org/en-US/packaging-guidelines/ .

PS: While I have above mentioned dpkg/rpm as examples to follow, it's not like those formats don't have problems either. https://nixos.org/nix/ and https://www.gnu.org/software/guix/ are perhaps the most prominent examples of 'next generation' packaging systems solving some of the problems of the old-school dpkg/rpm approaches.

Re: Pipenv: promises a lot, delivers very little

#170

Post by rch » Thu, Dec 06, 2018, 9:56 AM UTC

How did this get to be such a contentious issue? While I've been focused on other languages this year, my Python side projects have all gradually moved towards pipenv. I still have a few locally shared virtual environments, and setup.py is still the default for distribution. I also sometimes embed into uwsgi, but I don't know of anyone else that bothers with that. We're using pex at work.

Overall the article seems hyperbolic, relative to the actual events cited. Packaging is a work in progress everywhere.