Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python's New Package Landscape

andrewsforge.com

Post reply on HN
Print view View original

101–110 of 174 posts

Re: Python's New Package Landscape

#101

Post by xapata »

Serow225 wrote: post #81
patagonia wrote: post #60

Imagine I’m just getting started with Python, and I see this article. I think to myself, “Awesome, a primer!” Then I start reading (these comments)... mayyybe I should try Julia... or anything else, at least while I’m still getting started.

Does anyone know why the Python community has seemed to struggle with package mgmt fragmentation/churn so much over the years, compared to other languages? Did Guido just not really care about package mgmt?

Guido wrote Python in 1990. Did you use a package manager back then? I didn't.

Part of the issue is how well Python integrates with non-Python dependencies. Before conda, when I wanted to upgrade some Python projects, I'd get errors complaining about my Fortran compiler. These days, most of the major projects upload precompiled binaries for major platforms to PyPI, but when it was just source code...

Re: Python's New Package Landscape

#102

Post by neves »

prepend wrote: post #6
whitehouse3 wrote: post #3

While pipenv has garnered a lot of attention and praise for ease of use, it falls over whenever I integrate it with any serious work. Pipenv lock can take 20-30 minutes on a small flask app (~18 dependencies). And it often mixes up virtualenvs, enabling the wrong one with seemingly no remedy. I see the problems on Windows, MacOS and Ubuntu. 2018 is not the year of pipenv, for me. I'm sticking with regular virtualenvs…

Am I weird that I use anaconda instead of virtualenvs? I guess it’s overkill if you aren’t using the other conda features.

I think Conda is the way to go, because:

- there is Miniconda that doesn't force you to install all PyData packages. - virtual envs and needed packages are all defined in simple yaml file. - it works well with pip. So if a package isn't in Conda repository, you can install from pip. The annoyance here is that you must try conda, fail, and then try pip. - you can easily clone envs. So you can have some base envs with your usual packages (or one for Python 2 and another fo Python 3), and just clone them to start a new project.

Re: Python's New Package Landscape

#103

Post by rspeer »

lmm wrote: post #78
sambe wrote: post #55

Whenever talk in Python-world goes towards packaging, I feel like I have been transported to Javascript-world: it's never clear to me what concrete problems are being solved by the new tools/libraries. This article seems well-written and well-intentioned. Despite reading it, I don't know why I would not have loose dependencies in setup.py and concrete, pinned dependencies in requirements.txt. It's never felt hard to…

> loose dependencies in setup.py How does that work? How would someone else coming to work on your project use them? > concrete, pinned dependencies in requirements.txt How do you maintain that requirements.txt? And while that might work for applications, what do you do for libraries?

What would requirements.txt even _do_ in a library? AFAIK that file isn't even read when packaging, distributing, and installing libraries.

Re: Python's New Package Landscape

#104

Post by xapata »

aprdm wrote: post #91
patagonia wrote: post #76

Earlier quoted context omitted.

“Usually” is great until it’s not. I don’t want to get invested in a languages only to discover issues down the line. So, if something as basic as packaging is potentially problematic and there are other options available, I might look elsewhere before rolling the dice that this problem is not too problematic. Perhaps “Primer” was the wrong word, but I believe the sentiment is valid. Reading the comments, there is si…

pip and virtualenv has been solving the problem and is the standard for ever. other people tried different approaches, like, pipenv and is just that, a separate project trying to solve the same problem. I don't know why / who said that pipenv is the official recommended way, if it is it should not be and I hope it is not.

The Python community is not monolithic. PyPA has made some decisions that others might disagree with.

Re: Python's New Package Landscape

#105

Post by abalaji »

cik wrote: post #94

We just went through this cycle - ultimately we build packages (debs) and dockers, for deployment within VMs. Our build process - depending on the component pushes the deb to repos, or uses the deb in the docker. After trying to replace pip with Pipenv, we had to stop. The dependency resolution time for 20 declared dependencies (that in turn pull down > 100 components) takes well over 5 minutes. With poetry - it take…

I can second this... poetry is a seriously good project and matches PEP standards for the pyproject.toml, you will definitely see more projects jumping onto that standard soon.

Re: Python's New Package Landscape

#106

Post by rspeer »

hcrisp wrote: post #36
mlthoughts2018 wrote: post #27

Earlier quoted context omitted.

I’m disappointed your post did not cover using conda. As the pipenv drama has rolled on, I’ve moved from viewing conda merely as the best user experience in Python environment & package management to instead viewing it as the only serious option for professional scientific computuing work (and quite possibly any professional Python work at all).

Agree with you about using conda. And since no one has mentioned Jake Van der Plas' review of conda vs the alternatives, myths, etc., here it is: https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-mi...

I read that page and looked for the reason I don't use Conda (because I already have virtualenvs and I'm not prepared to burn them all down):

> Myth #5: conda doesn't work with virtualenv, so it's useless for my workflow

> Reality: You actually can install (some) conda packages within a virtualenv, but better is to use Conda's own environment manager: it is fully-compatible with pip and has several advantages over virtualenv.

> [...] the result seems to be fairly brittle – for example, trying to conda update python within the virtualenv fails in a very ungraceful and unrecoverable manner, seemingly related to the symlinks that underly virtualenv's architecture.

Doesn't sound like much of a myth then, if Conda's take on virtualenv is "you can technically do this, but everything will break ungracefully and unrecoverably, so please don't".

Re: Python's New Package Landscape

#107

Post by lmm »

rspeer wrote: post #103
lmm wrote: post #78

Earlier quoted context omitted.

> loose dependencies in setup.py How does that work? How would someone else coming to work on your project use them? > concrete, pinned dependencies in requirements.txt How do you maintain that requirements.txt? And while that might work for applications, what do you do for libraries?

What would requirements.txt even _do_ in a library? AFAIK that file isn't even read when packaging, distributing, and installing libraries.

Well presumably you want some way to have all the developers working on a library be using the same version of upstream libraries that that library depends on.

Re: Python's New Package Landscape

#108

Post by sambe »

lmm wrote: post #88
sambe wrote: post #80

Earlier quoted context omitted.

I assume that someone working on the project would do: pip install -e . in a virtual environment. I thought this was quite well-established. Is there a problem with it that I'm not aware of? pip freeze > requirements.txt for requirements.txt generation. For libraries just omit this? I'm not sure I understand the question. The article also mentions that several of the new tools aren't appropriate for libraries anyway.

> I assume that someone working on the project would do: pip install -e . in a virtual environment. I thought this was quite well-established. Is there a problem with it that I'm not aware of? So ignoring your requirements.txt, and potentially working with different versions of dependencies from the ones you were working with and encountering different bugs? (Also managing your virtual environments "by hand" is tedio…

People working on your project have the choice of using the requirements.txt or not. I would think core developers use the loose dependencies, with the aim of testing the latest and fixing the bugs. Someone has to move dependencies forward at some point, and doing this locally for knowledgeable people seems reasonable. CI should definitely - and part time contributors should probably - just use the pinned dependencies.

This is why I would not worry about pip freeze being non-reproducible. It is a manual step: upgrade our dependencies. Testing should happen all the time. If you are happy with the result of testing after upgrading dependencies, commit requirements.txt. I don't see new tools easing the burden of co-ordinating and testing dependency upgrades. Did I misunderstand them in this context?

I don't understand the concern for the library case. Pipenv doesn't address libraries. It seems to be an explicit goal of many people not to pin library dependencies. I'm asking what the new tools are solving - and again I can't see that they are solving this. Nothing is preventing you from pinning your library dependencies if you want (using old tools) but you'll probably get people complaining about being incompatible with other projects.

Re: Python's New Package Landscape

#109

Post by cik »

abalaji wrote: post #105
cik wrote: post #94

We just went through this cycle - ultimately we build packages (debs) and dockers, for deployment within VMs. Our build process - depending on the component pushes the deb to repos, or uses the deb in the docker. After trying to replace pip with Pipenv, we had to stop. The dependency resolution time for 20 declared dependencies (that in turn pull down > 100 components) takes well over 5 minutes. With poetry - it take…

I can second this... poetry is a seriously good project and matches PEP standards for the pyproject.toml, you will definitely see more projects jumping onto that standard soon.

That was the part that made me try it, literally while waiting for Pipenv to resolve.

Re: Python's New Package Landscape

#110

Post by epage »

siquick wrote: post #45
epage wrote: post #37

The biggest problem I have with python packaging tools is how do I start using them. I'd rather not install all of them in my global site-packages. Do I need to create a virtualenv just to get a tool to manage my virtualenv's? I have seen poetry is working on their bootstrapping story. I could not get their current solution to work on Ubuntu. Maybe what they are developing towards will work. https://github.com/sdispa…

pipenv --three && pipenv shell && pipenv install and you're set

Except how do I install pipenv in the first place?

Or why do I need to use pipenv to install pipsi so pipsi can manage my virtualenv environments.

Post reply on HN