Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python's New Package Landscape

andrewsforge.com

Post reply on HN
Print view View original

21–30 of 174 posts

Re: Python's New Package Landscape

#21

Post by hultner »

I've migrated to pipenv in most of my projects, it's simple and great for application development but I still write everything to work with pure pip as well so the Pipfile basically lists my application as a dependency and I mainly use it for the lock files.

For library development I target pure pip/setuptools but still use pipenv during development phase. There have been a few cases where pipenv had problems and I had to either remove my virtualenv and reinitialize it or even remove my pip-file/lockfile, but since I still have my setup.py it's not a big deal for me.

As for uploading etc I use twine but I wrap everything in a makefile to make handling easier.

A problem I noticed recently was a case where one of my developers used a tool which was implicitly installed in the testing environment since it was a subdependency of a testing tool but it was not installed into the production image. This resulted in "faulty" code passing the CI/CD and got automatically deployed to the live development environment where it broke (so it never reached staging). Caused a little bit of a headache before I found the cause.

Re: Python's New Package Landscape

#22

Post by azag0 »

For pure-python library projects, I found Poetry the best option these days (haven't tried Hatch). But it is still heavily under development, so it's not necessarily a black-box solution.

The biggest pain point of Pipenv for me is that it cannot as yet selectively update a single dependency without updating the whole environment.

Re: Python's New Package Landscape

#23

Post by rhizome31 »

samwillis wrote: post #8

Not really packaging but related, my favourite new tool is Pyenv ( https://github.com/pyenv/pyenv ) it made getting a new laptop setup with various versions of Python so much quicker. I haven’t used Pipenv yet but it works with pyenv to create virtual envs with a specified puthon version as well as all the correct packages.

Polyglots might also be interested in asdf (https://github.com/asdf-vm/asdf). It's like Pyenv but supports various languages via a plugin system (eg. https://github.com/danhper/asdf-python).

Re: Python's New Package Landscape

#25

Post by whitehouse3 »

scrollaway wrote: post #15
whitehouse3 wrote: post #3

While pipenv has garnered a lot of attention and praise for ease of use, it falls over whenever I integrate it with any serious work. Pipenv lock can take 20-30 minutes on a small flask app (~18 dependencies). And it often mixes up virtualenvs, enabling the wrong one with seemingly no remedy. I see the problems on Windows, MacOS and Ubuntu. 2018 is not the year of pipenv, for me. I'm sticking with regular virtualenvs…

> Pipenv lock can take 20-30 minutes on a small flask app (~18 dependencies) Do you have scipy/numpy/keras or cython somewhere in the deps? pipenv lock is slow, but not 20-30 mins slow unless there's a very very large download and/or a long compilation somewhere in there.

The web app in question depends on PANDAS + numpy so that's definitely part of the toolchain. It wasn't 20-30 minutes from day one. The lock time started fast and then ballooned. Other comments here saying 2-3 minutes per lock are consistent with my general experience.

This wasn't for complex pipenv operations either. A simple command: pipenv run python main.py took progressively longer to execute.

Re: Python's New Package Landscape

#26

Post by phren0logy »

I'm not a programmer by trade, but I dabble, and these issues make it much less fun.

In my limited experience, Clojure's Leinengen is a far more pleasant way to solve these problems. I'm sure there are many other examples in other languages, but in the few I've used, nothing comes close. Each project has versioned dependencies, and they stay in their own little playground. A REPL started from within the project finds everything. Switch directories to a different project, and that all works as expected, too. It's a dream.

[https://leiningen.org/]

Re: Python's New Package Landscape

#27

Post by mlthoughts2018 »

Keats wrote: post #18
whitehouse3 wrote: post #3

While pipenv has garnered a lot of attention and praise for ease of use, it falls over whenever I integrate it with any serious work. Pipenv lock can take 20-30 minutes on a small flask app (~18 dependencies). And it often mixes up virtualenvs, enabling the wrong one with seemingly no remedy. I see the problems on Windows, MacOS and Ubuntu. 2018 is not the year of pipenv, for me. I'm sticking with regular virtualenvs…

I tried a few of the package management in Python recently ( https://www.vincentprouillet.com/blog/overview-package-manag... ) and had the same conclusion with Pipenv. It is way too slow and frankly the UX is not that great either.

I’m disappointed your post did not cover using conda. As the pipenv drama has rolled on, I’ve moved from viewing conda merely as the best user experience in Python environment & package management to instead viewing it as the only serious option for professional scientific computuing work (and quite possibly any professional Python work at all).

Re: Python's New Package Landscape

#28

Post by Bogdanp »

rhizome31 wrote: post #23
samwillis wrote: post #8

Not really packaging but related, my favourite new tool is Pyenv ( https://github.com/pyenv/pyenv ) it made getting a new laptop setup with various versions of Python so much quicker. I haven’t used Pipenv yet but it works with pyenv to create virtual envs with a specified puthon version as well as all the correct packages.

Polyglots might also be interested in asdf ( https://github.com/asdf-vm/asdf ). It's like Pyenv but supports various languages via a plugin system (eg. https://github.com/danhper/asdf-python ).

For those wondering (like I did): it doesn't seem like this has any relation to Common Lisp's ASDF.

Re: Python's New Package Landscape

#29

Post by 131012 »

azag0 wrote: post #16
prepend wrote: post #6

Earlier quoted context omitted.

Am I weird that I use anaconda instead of virtualenvs? I guess it’s overkill if you aren’t using the other conda features.

It is an overkill for pure-python packages or packages with simple C extensions. Conda was developed specifically to handle non-python dependencies, which would be difficult to build in setup.py. Also, a conda package is not a replacement for a distutils/setuptools package. When building a conda package, one still calls setup.py. So every python conda package has to be a distutils/setuptools package anyway.

Thanks for caveat. Nonetheless, anaconda makes my life so much easier when working with python libraries. If anybody got any other reasons to be careful of it, I'm interested!

Re: Python's New Package Landscape

#30

Post by scrollaway »

Bogdanp wrote: post #19
scrollaway wrote: post #15

Earlier quoted context omitted.

> Pipenv lock can take 20-30 minutes on a small flask app (~18 dependencies) Do you have scipy/numpy/keras or cython somewhere in the deps? pipenv lock is slow, but not 20-30 mins slow unless there's a very very large download and/or a long compilation somewhere in there.

It takes about 2 minutes (feels like 5!) on my 2016 MBP to install 102 dependencies. Doing that in Docker takes about 1.5x the time. I haven't seen it take 20-30 minutes, but 2-3 minutes is still obscenely slow in my view.

A lot of this time may be spent on downloading the dependencies to the cache. If you're doing it in docker, you likely don't have a persistent cache. I've hit this issue before. https://github.com/pypa/pipenv/issues/1785

If you configure the cache properly you might solve it, but yeah it's kinda dumb it has to do that just for locking.

Post reply on HN