Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python's New Package Landscape

andrewsforge.com

Post reply on HN
Print view View original

121–130 of 174 posts

Re: Python's New Package Landscape

#121

Post by guitarbill »

kankroc wrote: post #47
whitehouse3 wrote: post #3

While pipenv has garnered a lot of attention and praise for ease of use, it falls over whenever I integrate it with any serious work. Pipenv lock can take 20-30 minutes on a small flask app (~18 dependencies). And it often mixes up virtualenvs, enabling the wrong one with seemingly no remedy. I see the problems on Windows, MacOS and Ubuntu. 2018 is not the year of pipenv, for me. I'm sticking with regular virtualenvs…

Can you elaborate on why using a requirements.txt file is "manual hell"? I rely on it for pretty much everything and I didn't run into game breaking problems.

because if you pin stuff in requirements.txt, they either never get updated, or you have to go through, check which ones have updated, and manually edit the requirements.txt. the combination of Pipfile and Pipfile.lock were designed to solve this in a much better way (briefly: understanding standard deps vs development deps, and using the Pipfile.lock file for exact pinning/deployment pinning, vs general compatibility pinning in the Pipfile).

Re: Python's New Package Landscape

#123

Post by craftsman »

aprdm wrote: post #91
patagonia wrote: post #76

Earlier quoted context omitted.

“Usually” is great until it’s not. I don’t want to get invested in a languages only to discover issues down the line. So, if something as basic as packaging is potentially problematic and there are other options available, I might look elsewhere before rolling the dice that this problem is not too problematic. Perhaps “Primer” was the wrong word, but I believe the sentiment is valid. Reading the comments, there is si…

pip and virtualenv has been solving the problem and is the standard for ever. other people tried different approaches, like, pipenv and is just that, a separate project trying to solve the same problem. I don't know why / who said that pipenv is the official recommended way, if it is it should not be and I hope it is not.

> I don't know why / who said that pipenv is the official recommended way, if it is it should not be and I hope it is not.

The Python Packaging Authority says that pipenv is the first "recommended way" for managing application dependencies: https://packaging.python.org/guides/tool-recommendations/#ap...

Except if pipenv doesn't meet your needs. Then use pip.

Or, if you need cross-platform support, use buildout.

Or, if you are doing scientific computation, don't use any of those, use conda, Hashdist, or Spack.

Or if you need to create a package, use setuptools and twine.

So, no, pip and virtualenv don't solve the problem, because there are a lot of different problems and use-cases. I can say from my experience that conda is _the_ best solution to the problem for scientific work.

Re: Python's New Package Landscape

#124

Post by zmmmmm »

patagonia wrote: post #60

Imagine I’m just getting started with Python, and I see this article. I think to myself, “Awesome, a primer!” Then I start reading (these comments)... mayyybe I should try Julia... or anything else, at least while I’m still getting started.

Wait until someone from the python community recites to you "there should be one and only one good way to do something" ....!

Re: Python's New Package Landscape

#125

Post by hcrisp »

rspeer wrote: post #106
hcrisp wrote: post #36

Earlier quoted context omitted.

Agree with you about using conda. And since no one has mentioned Jake Van der Plas' review of conda vs the alternatives, myths, etc., here it is: https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-mi...

I read that page and looked for the reason I don't use Conda (because I already have virtualenvs and I'm not prepared to burn them all down): > Myth #5: conda doesn't work with virtualenv, so it's useless for my workflow > Reality: You actually can install (some) conda packages within a virtualenv, but better is to use Conda's own environment manager: it is fully-compatible with pip and has several advantages over vi…

He's not saying that you _should_ install conda within a virtualenv, but that some have tried with some success.

At the end, one of his conclusions is: "If you want to install Python packages within an Isolated environment, pip+virtualenv and conda+conda-env are mostly interchangeable". So don't change if you don't have to.

But he does give reasons why conda may be superior to virtualenv -- managing different version of Python, tracking non-Python dependencies, true isolation of environments, etc.

Re: Python's New Package Landscape

#126

Post by ubercore »

whitehouse3 wrote: post #3

While pipenv has garnered a lot of attention and praise for ease of use, it falls over whenever I integrate it with any serious work. Pipenv lock can take 20-30 minutes on a small flask app (~18 dependencies). And it often mixes up virtualenvs, enabling the wrong one with seemingly no remedy. I see the problems on Windows, MacOS and Ubuntu. 2018 is not the year of pipenv, for me. I'm sticking with regular virtualenvs…

Something sounds broken here, I have projects with similar numbers of dependancies, including heavy ones like pandas and numpy, and don't get anywhere near that long to lock. I don't have a specific suggestion for you, though. How long does a regular `pip install -r requirements.txt` take for the same dependancies?

Re: Python's New Package Landscape

#127

Post by ihumanable »

abalaji wrote: post #105
cik wrote: post #94

We just went through this cycle - ultimately we build packages (debs) and dockers, for deployment within VMs. Our build process - depending on the component pushes the deb to repos, or uses the deb in the docker. After trying to replace pip with Pipenv, we had to stop. The dependency resolution time for 20 declared dependencies (that in turn pull down > 100 components) takes well over 5 minutes. With poetry - it take…

I can second this... poetry is a seriously good project and matches PEP standards for the pyproject.toml, you will definitely see more projects jumping onto that standard soon.

Sébastien Eustace is a really awesome developer, I've used Pendulum and Orator as well as Poetry for projects, the documentation is beautiful and complete, the APIs are well thought out, and if you find an issue contributing back to the project is simple and straightforward (I'm happy to have a few PRs accepted into Orator).

There's a certain joy working with tools when it's clear that the person making those tools actually cares about the developer and making it work well.

Re: Python's New Package Landscape

#128

Post by frafra »

guitarbill wrote: post #121
kankroc wrote: post #47

Earlier quoted context omitted.

Can you elaborate on why using a requirements.txt file is "manual hell"? I rely on it for pretty much everything and I didn't run into game breaking problems.

because if you pin stuff in requirements.txt, they either never get updated, or you have to go through, check which ones have updated, and manually edit the requirements.txt. the combination of Pipfile and Pipfile.lock were designed to solve this in a much better way (briefly: understanding standard deps vs development deps, and using the Pipfile.lock file for exact pinning/deployment pinning, vs general compatibilit…

Btw, it is possible to use the compatible ~= operator (PEP 440) within requirements.txt.

Re: Python's New Package Landscape

#129

Post by theptip »

kankroc wrote: post #47
whitehouse3 wrote: post #3

While pipenv has garnered a lot of attention and praise for ease of use, it falls over whenever I integrate it with any serious work. Pipenv lock can take 20-30 minutes on a small flask app (~18 dependencies). And it often mixes up virtualenvs, enabling the wrong one with seemingly no remedy. I see the problems on Windows, MacOS and Ubuntu. 2018 is not the year of pipenv, for me. I'm sticking with regular virtualenvs…

Can you elaborate on why using a requirements.txt file is "manual hell"? I rely on it for pretty much everything and I didn't run into game breaking problems.

It requires a lot of work to produce reproducible/secure builds, see the original Pipfile design discussion for gory details:

https://github.com/pypa/pipfile

Re: Python's New Package Landscape

#130

Post by jonafato »

Serow225 wrote: post #119
xapata wrote: post #101

Earlier quoted context omitted.

Guido wrote Python in 1990. Did you use a package manager back then? I didn't. Part of the issue is how well Python integrates with non-Python dependencies. Before conda, when I wanted to upgrade some Python projects, I'd get errors complaining about my Fortran compiler. These days, most of the major projects upload precompiled binaries for major platforms to PyPI, but when it was just source code...

Sure, but couldn't it have been given some BFDL/high-level leadership importance in the last five years to rein in the craziness?

The PyPA team has done a lot over the past five years. The changelog for pip (https://pip.pypa.io/en/stable/news/) contains quite a bit, PyPI was migrated to Warehouse, and there have been several PEPs focused on improving the packaging situation. A lot of these ideas come from various people in the community and get formalized as official recommendations or tools, but these things take time, especially accounting for backward compatibility in an ecosystem as large and mature as Python's.

The short answer to "why isn't this solved?" is "it's hard, and there's a lot to do". Development practices change over time, and the tooling continues to evolve with them. It's easy to see a broad survey like this and think that there's too much going on, but taken at a high level, the space is definitely trending in the right direction.

(Note: I'm not part of the PyPA, but I'm interested in this area and try to follow along from the outside.)

Post reply on HN