Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Overview of Python dependency management tools

modelpredict.com

Post reply on HN
Print view View original

131–140 of 184 posts

Re: Overview of Python dependency management tools

#131

Post by acomjean »

dfsegoat wrote: post #115
acomjean wrote: post #80

Earlier quoted context omitted.

I think it depends on the use case. If I'm developing my own stuff my peen package management is fine. If trying to run various existing python programs to analyze biology data, I soon run into various problems. Is this a Conda?/ or can I use my Python environment? which version of python? will let me run the thing and what libraries do I need? This breaks in that version? Sometimes I feel that one kinda ok way of do…

> Is this a Conda?/ or can I use my Python environment? Can you elaborate a bit there? I use conda because I like some of their features over standard virtualenv (being able to specify a python version when i create my venv) - but I've never had a problem running code in env's created by one vs. the other.

Sometimes developers distribute their software via Conda installs, in those cases they sometimes don't provide instructions on running other ways (eg using the pyenv environment which is my default. ). I'm ok with this, but when conda fails as sometimes happens, it can mean some digging to get the install to work.

I was thinking of my latest install, which was CRISPRESSO2, which installs via docker or bioconda... I was able to get it going, but it took a bit on some systems.. (Python 2.7 old libraries.. etc..) Docker didn't seem to work.

I like virtual env, but sometimes I feel I have to have a new environment for each piece of software I'm running, which feels weird.

https://github.com/pinellolab/CRISPResso2

Re: Overview of Python dependency management tools

#132

Post by postpawl »

ploxiln wrote: post #105
postpawl wrote: post #35

Earlier quoted context omitted.

I think that will end up installing the subdependency version of whatever is last in the requirements.txt. You need a dependency resolver to deal with problems with conflicting versions. More details here: https://medium.com/knerd/the-nine-circles-of-python-dependen...

pip handles the simple cases: if you install a new pkgA that depends on 'pkgB =3'. You probably want for pip (or similar) to figure out that an older version of 'pkgC' is compatible with 'pkgB>2'. But I actually don't want it to be too smart. Better to keep your dependencies minimal and explicit, and manually specify older 'pkgC' if you need to. I have a few non-trivial services in production, the most complex one wi…

The problem with doing things this way is that you’re not going to know there’s a problem until there’s an issue in your tests (hopefully) or production. You’ll eventually install something new, it’ll update some subdependencies to a version that another library doesn’t support, and then things get broken. Pip-tools is easy to use and it tells you there’s a problem before it’s too late.

Re: Overview of Python dependency management tools

#133

Post by sergeykish »

a_cool_username wrote: post #116
sergeykish wrote: post #107

Earlier quoted context omitted.

Looks like any other package manager: * developers install with language packager * in between install with OS package manager * users install bundle Those who have troubles with pip, gems, cabal, etc should check over options first. Wait, bundlers Gemfile.lock lists installed versions at least ten years, what is "too unbounded" in pip?

It depends! Sometimes I have to lock a dependency at minor releases because every.single.release from the author breaks something new, and I've already worked around the locked version's failings. Sometimes I have to lock a dependency at a major version and everything is fine after that. Usually when the latter happens, eventually the developer releases something that fits within the version bounds and breaks. Someti…

I've recently reported bug on Xmonad github, they have

### Checklist

  - [ ] I've read [CONTRIBUTING.md](https://github.com/xmonad/xmonad/blob/master/CONTRIBUTING.md)

  - [ ] I tested my configuration with [xmonad-testing](https://github.com/xmonad/xmonad-testing)
I think it is brilliant idea, immediately checked latest git versions, I assume you may add

  - [ ] I tested my application with [latest stable requirements.txt](...)
And something about triangulation and reporting to another repo too.

Sorry to hear about breaks on major version. Ruby gems (libraries) freeze dependencies on major, sometimes minor, example [0]. But applications shipped with Gemfile and Gemfile.lock [1], [2]. So `bundle install` is reproducible [3]:

> The presence of a `Gemfile.lock` in a gem's repository ensures that a fresh checkout of the repository uses the exact same set of dependencies every time. We believe this makes repositories more friendly towards new and existing contributors. Ideally, anyone should be able to clone the repo, run `bundle install`, and have passing tests. If you don't check in your `Gemfile.lock`, new contributors can get different versions of your dependencies, and run into failing tests that they don't know how to fix.

Yes, docker, msi, Flatpack, AppImage - whatever works for you and your users. It is sad we can't easily statically compile in one file on scripting languages.

[0] https://github.com/teamcapybara/capybara/blob/master/capybar...

[1] https://github.com/Shopify/example-ruby-app/blob/master/Gemf...

[2] https://github.com/Shopify/example-ruby-app/blob/master/Gemf...

[3] https://bundler.io/guides/faq.html

Re: Overview of Python dependency management tools

#134

Post by WD-42 »

madelyn wrote: post #50

People always get up in arms about this, but as someone who has used Python as her daily driver for years it's really... never been this serious of an issue for me? I have used virtualenv/venv and pip to install dependencies for years and years, since I was a teen hacking around with Python. Packaging files with setup.py doesn't really seem that hard. I've published a few packages on pypi for my own personal use and…

The only time I run into problems is when someone else is trying to use Conda. Then it can be hell trying to get their code running in standard pip/venv or vice versa.

I'm sure Anacona filled a niche at some point, but we have wheels now, can we all just agree to stop using Conda? What value does it actually bring now that makes it worth screwing up the standard distribution tools?

Re: Overview of Python dependency management tools

#135

Post by sixhobbits »

I've spent far too many hours fighting with these tools in two completely different scenarios

* Developing and deploying production Python solutions

* Helping beginners run their first script

While it's great for beginners to use the same tools that are used in industry, I strongly believe that the problem nearly all of these tools face is that they can't decide whether they want to _manage_ complexity or _hide_ complexity.

You can't do both.

Some of them do a fairly good job at managing complexity. None of them do a good job of hiding it. The dream of getting Python to "just work" on any OS is close to impossible (online tools like repl.it are the closest I've found but introduce their own limtiations). I recently saw a place force their beginner students onto Conda in Docker because getting people started with Conda was too hard. If you're battling with the complexity of your current layer of abstraction, sometimes it's better to start removing abstraction rather than adding more.

That said, I'm also a happy user of `pip` and `virtualenv` and while I'm sure that many people can use the others for more specific needs, I think defaulting to them because they aim to be "simpler" is nearly always a mistake. I still teach beginners to install packages system wide without touching venv at first - it's enough to get you through your first 2-3 years of programming usually.

Re: Overview of Python dependency management tools

#136

Post by alxmdev »

Reading this makes me grateful that I can get away with just apt-get. I wonder how prevalent this is, since not every project needs specific or latest versions of the runtime and libraries, only a minimum. Some are just plumbing tools that stick to the stable core, and the Python 3 ecosystem has been mature for enough years that older distro packages are still useful and capable.

Re: Overview of Python dependency management tools

#138

Post by franey »

kissgyorgy wrote: post #119

> Pipenv or poetry? If you used pipenv for a complex project with huge dependency tree, or used it for a long time, you definitely run into a blocker issue with it. That is the worst package manager of all, and probably the reason why Python has such a bad reputation in this area. It's because it's fundamentals are terrible. Just go with Poetry. It's very stable, easy to use, has a superior dependency resolver and wa…

Pipenv has had a couple releases recently, but I've had an easier time with Poetry. Poetry is almost always[0] faster than Pipenv, and I find its commands more intuitive.

I've been meaning to take another look at Pipenv, but the huge pause without a release makes me nervous that it could happen again.

[0] https://johnfraney.ca/posts/2019/11/19/pipenv-poetry-benchma...

Re: Overview of Python dependency management tools

#139

Post by ptx »

a_cool_username wrote: post #104
ptx wrote: post #92

Earlier quoted context omitted.

> struggling to get Python installed and into their `PATH` ... it's frankly still amateur hour over there But that has been solved on Windows for quite a while hasn't it? Python installs the "py" launcher on the path, which allows you to run whichever version you want of those you have installed. Just type "py" instead of "python". Or "py -3.5-32" to specifically run 32-bit Python 3.5, or "py -0" to list the availabl…

It's gotten a lot better, but we still hit tons of issues with users who don't know what Python version they installed their application in. Oh and of course our "binaries" in Scripts/bin don't seem to show up in the PATH by default. So I get to tell people "py -3.8-64 -m foo" on windows, "foo" everywhere else. This gets much much worse when a new version of Python comes out and we don't support it yet (because of th…

Sure, but telling people to run "py -3.7" seems a lot easier than walking them through uninstalling and reinstalling Python, as you would have had to in the bad old days. It's reliable and consistent and doesn't depend of what's installed where or how it's configured. If you run "py -3.7 -m venv my_env", it just works, always, with no special context required.

Although I don't handle user support for Python packages, if I did, that would be my go-to approach.

Re: Overview of Python dependency management tools

#140

Post by franey »

WillDaSilva wrote: post #21

This is a good overview of something that took me an annoyingly long time to learn. My personal preference is to keep things simple with pyenv, venv, and pip. Tangentially related is the tool tox [1], which is often used to run a test suite inside of virtual environments created by venv, on multiple versions of Python managed by pyenv. Now if only setuptools could work well without hackery... [1]: https://tox.readthe…

Poetry does a good job of managing/publishing a package without having to custom-code a setup.py. I have a blog post about it but I don't want to spam. Poetry works well for a package of average complexity, and it's configured entirely in pypyroject.toml. I don't have experience publishing something complicated with it, though.
Post reply on HN