How are you ever going to manage Python Packages? PyPi and pip
1–10 of 17 posts
Re: How are you ever going to manage Python Packages? PyPi and pip
#2This article is missing the best tool of the bunch: virtualenv. Part of the fundamental reason managing python packages is such a pain (or perl packages, or ruby packages, etc) is that they're installed once for the whole system.
Virtualenv allows you to sandbox all your packages in a directory local to your development/deployment environment. You no longer need to install anything on the base system, and you can have multiple virtualenvs side-by-side. In addition, you can copy a virtualenv from your build environment to your production environment by just copying the directory.
This gives you huge wins in versioning packages, testing out different versions alongside each other, and being able to strongly validate that what you tested in your staging environment is the same thing that got deployed to your production systems.
In general, I'm a very strong believer of pushing _everything_ your production environment relies on to a single directory on your production system, rather than installing it on the base system. It's actually rather surprising more people haven't moved to this model, and that there aren't better tools to support it.
Re: How are you ever going to manage Python Packages? PyPi and pip
#3I had a rather less flattering experience. It seems half the time I pull stuff using pip, it will fetch the wrong version or fail to compile, or not fetch essential dependencies...
It is still good, but nowhere near as great as gem or apt. But maybe that is just me being unlucky...
Re: How are you ever going to manage Python Packages? PyPi and pip
#4Didn't know about requirement files. Useful!
Re: How are you ever going to manage Python Packages? PyPi and pip
#5This article is missing the best tool of the bunch: virtualenv. Part of the fundamental reason managing python packages is such a pain (or perl packages, or ruby packages, etc) is that they're installed once for the whole system. Virtualenv allows you to sandbox all your packages in a directory local to your development/deployment environment. You no longer need to install anything on the base system, and you can hav…
you can copy a virtualenv from your build environment to your production environment by just copying the directory
Not if you're developing on a different platform/architecture than you're deploying to (e.g. OS X vs. Linux). Unless you're only using pure Python modules, which excludes PIL and most database drivers.
Other than that, you're very right.
Re: How are you ever going to manage Python Packages? PyPi and pip
#6This article is missing the best tool of the bunch: virtualenv. Part of the fundamental reason managing python packages is such a pain (or perl packages, or ruby packages, etc) is that they're installed once for the whole system. Virtualenv allows you to sandbox all your packages in a directory local to your development/deployment environment. You no longer need to install anything on the base system, and you can hav…
Hi, I'm the author of the article. We've definitely looked at virtualenv but for right now we're just developing a single app. Thus far, we've been able to get away with munging the global package list but this could change in the near future. I tried playing around with virtualenv earlier this year but I couldn't figure out how to push directories OR make virtualenvs reliably relocatable. The main problems for us were/are:
1) We develop on macs 2) Our deploy environments are a mix of 32-bit/64-bit machines running Ubnutu 3) It wasn't clear if each deployment required its own virtualenv to start from scratch OR if we should reuse a virtualenv (which seemed to defeat the whole point of using virtualenv)
Got any ideas on what we could do?
Re: How are you ever going to manage Python Packages? PyPi and pip
#7I had a rather less flattering experience. It seems half the time I pull stuff using pip, it will fetch the wrong version or fail to compile, or not fetch essential dependencies... It is still good, but nowhere near as great as gem or apt. But maybe that is just me being unlucky...
Often these are issues with improper setup.py configurations, by the authors of these packages. It's not uncommon for authors to forget to list their package dependencies in setup.py, for instance.
Forking these projects and fixing the setup.py is probably the best solution. If they're active projects and you'll want their updates, they're likely to accept your patch. If they're inactive, it doesn't really matter that you're using a fork anyway.
Re: How are you ever going to manage Python Packages? PyPi and pip
#8I had a rather less flattering experience. It seems half the time I pull stuff using pip, it will fetch the wrong version or fail to compile, or not fetch essential dependencies... It is still good, but nowhere near as great as gem or apt. But maybe that is just me being unlucky...
You can explicitly specify which version of a package you want to install. For example, if you wanted to install an older version of Tornado,
pip install tornado==1.1
OR if you know you ONLY want tornado 2.0
pip install tornado==2.0
From my experience, pip has been able to fetch PYTHON dependencies if the author of the package wrote their setup.py correctly. I agree that things could be better with regards to C dependencies (MySQL-Python requiring libmysql5dev being particularly annoying) but this is hard to do as every environment has their own way of naming dependent libs.
Re: How are you ever going to manage Python Packages? PyPi and pip
#9This article is missing the best tool of the bunch: virtualenv. Part of the fundamental reason managing python packages is such a pain (or perl packages, or ruby packages, etc) is that they're installed once for the whole system. Virtualenv allows you to sandbox all your packages in a directory local to your development/deployment environment. You no longer need to install anything on the base system, and you can hav…
Hi, I'm the author of the article. We've definitely looked at virtualenv but for right now we're just developing a single app. Thus far, we've been able to get away with munging the global package list but this could change in the near future. I tried playing around with virtualenv earlier this year but I couldn't figure out how to push directories OR make virtualenvs reliably relocatable. The main problems for us we…
I'm relatively new to using virtualenv myself, but I think you replicate a virtualenv on a different os, architecture or location by creating a new virtualenv on the target machine then using pip to reinstall all the packages. You use "pip freeze" in your source environment to generate the package list then "pip install --requirements=" to reinstall packages on the target.
Re: How are you ever going to manage Python Packages? PyPi and pip
#10If you target a particular Linux platform and have more than just pure Python package dependenices it might be better to learn to use that distro's native package manager (RPM for example).
Not sure if pip lets you run configuration scripts during various stages of install and un-install?
Does pip let you depend on C libraries and other system features?
How does pip handle obsoletes and transitive dependencies?