Just a thank you for writing a positive, easy to follow overview with links to more in-depth information. I love when people boil down experience and serve it without a side dish of attitude.
Python Deployment Anti-Patterns
61–70 of 125 posts
Re: Python Deployment Anti-Patterns
#62Earlier quoted context omitted.
The worst thing about them is the fact that they are installed into global site-packages that you shouldn’t use for any serious coding. And yes, they are mostly outdated too.
> "into global site-packages that you shouldn’t use for any serious coding" Any specific reason for that? I find it quite good and have quite large deployments using .debs only with packages in global location. (tens of packages produced locally - either updated or unavailable dependencies and the service itself) Any direct dependency is handled by package pinning and no update goes into production untested, so the w…
You never have two apps deployed on the same server that need different versions of a dependency?
Re: Python Deployment Anti-Patterns
#63Earlier quoted context omitted.
I think there's an exageration as well, today the trend seems to be "use nothing by the distro" Ok, sure, MongoDB still changes a lot between versions, in this case you should use the latest version. But stop there. Especially if you're paying for support (like RHEL) There should be a good reason for you to compile Apache / MySQL / PostgreSQL / Python. Otherwise, use the distro version. One (common) exception would b…
Yes you should, but if you have them: do it. That’s what the article says. ;) Please don’t push it in a wrong direction. What I actually wrote is: because we’re a LAMP web hoster, we compile MySQL+PHP+Apache ourself. And because we’re a Python shop, we don’t let Ubuntu/Red Hat dictate which Python version we use.
Some choices certainly make more sense for web hosters than run-of-the-mill Python dev shops.
Re: Python Deployment Anti-Patterns
#64We do this, but also add a couple layers of safety between us and PyPI: 1. Run your own secure, local pypi clone with exact source versions of the packages you use. 2. The packages for production are built into RPMs from the local pypi. PyPI is great for discovery, getting things running quickly, and testing new versions, but you never want to rely on it, even for development.
I'd really enjoy reading about how to setup a PyPI mirror like the one you use in your development/deployment workflow. It seems like a really good idea, considering I've had problems with PyPI at really inconvenient times in the past.
I've used ClueReleaseManager to great effect in the past. It lazy loads from an upstream PyPI so you don't need to maintain a full mirror. If you use the same instance as your main PyPI mirror for developing and CI/deployment you never have to worry about syncing manually: any package you develop with will be in the cache by the time your code hits the CI server.
Re: Python Deployment Anti-Patterns
#65Earlier quoted context omitted.
Fabric seems to be the most popular deployment tool, yet the author advises against it without giving any reason why. I'd love to see some proper detail in the article around why. And not in the vein of "Chef/Puppet are better", but more along the line of "here's what can go wrong with Fabric".
Fabric is an invaluable deployment tool, but you have to know its boundaries for what it was intended. With Fabric you tell what to _do_ and with Puppet/Chef you define what the result should _look_ like. You define how a server should look like and it can make sure its true for 1000 servers. Or 10000. It’s not about Fabric vs. Puppet, but how to use both in the most efficient way.
I find the line between when to use fabric and when to use puppet/chef is needing the permissions of a more privileged user.
I prefer to deploy applications in the home directory of a dedicated user account with minimal privileges, and use fabric for installing updates and running application tasks.
OTOH, I'd prefer to be using puppet for creating the user accounts, managing the installation and configuration of PostgreSQL, putting app configuration in a safe location, and so on.
This comes down to my (maybe antiquated?) view of having multiple applications running on a single server.
Re: Python Deployment Anti-Patterns
#66Earlier quoted context omitted.
I would disagree with this. LTS (I'm referring to Ubuntu LTS as well as RHEL) is still a good fit because there are 9001 other packages your system relies on for day-to-day operations, and I'd rather mess with those as least often as possible. If you don't use a LTS then you'll have to do whole system upgrades every few years - or else you stop getting security backports. I'd much rather micromanage the dozen highly…
there's still a chance that you'll be stuck with the just-before-latest-stable version. Is that so bad? Does python really change that much from release to release?
You don't really want to be on anything earlier than 2.7 right now.
Re: Python Deployment Anti-Patterns
#67We do this, but also add a couple layers of safety between us and PyPI: 1. Run your own secure, local pypi clone with exact source versions of the packages you use. 2. The packages for production are built into RPMs from the local pypi. PyPI is great for discovery, getting things running quickly, and testing new versions, but you never want to rely on it, even for development.
I'd really enjoy reading about how to setup a PyPI mirror like the one you use in your development/deployment workflow. It seems like a really good idea, considering I've had problems with PyPI at really inconvenient times in the past.
I think we use this: http://pypi.python.org/pypi/chishop/
You just set it up on a local server, and upload packages the same way they are uploaded to real PyPI.
python setup.py register sdist upload
Re: Python Deployment Anti-Patterns
#68I am glad this is working for the OP, but pushing virtualenv and "self-contained" apps as the one solution is a diservice to the community. There are valid reasons to rely on your OS, assuming you have an homogenous deployment target (same OS, maybe different versions):
- lots of people argue for virtualenv because some versions may be incompatible. The problem here is the lack of backward compatibility of packages, and frankly, if you need to rely on packages which change API willingnily betwen e.g. 1.5 and 1.6, or if each of your service depends on a different version of some library, you have bigger problems anyway.
- any sufficiently complex deployment will depend on things that are not python, at which point you need a solution that integrates multiple languages. That is, you re-creating what a distribution is all about.
- virtualenv relies on sources, so if some of your dependences are in C, every deploy means compilation
- I still have no idea how security is handled when you put everything in virtualenv
See also http://bytes.com/topic/python/answers/841071-eggs-virtualenv...
Re: Python Deployment Anti-Patterns
#69Virtualenv is a half solution and a hack. Use vagrant and VMs. There's a whole sea of libs and software that isn't "versioned" by pip/virtualenv.
supervisord is the wrong solution. It answers the wrong question (is the process running). It's worse than useless in that it has given false positives. The right question is (is the process responding correctly). Use monit or something else that actually does what's needed.
Re: Python Deployment Anti-Patterns
#70Earlier quoted context omitted.
Yes to all of this and I'll throw in my two cents on our web server setup which is nginx + uwsgi (which has served remarkably well).
Likewise +1 to Nginx + uwsgi. In the various performance benchmarks* uwsgi beats gunicorn, and uwsgi comes bundled with Nginx now, so there's really no reason to bother with setting up gunicorn any more. * http://www.peterbe.com/plog/fcgi-vs-gunicorn-vs-uwsgi http://nichol.as/benchmark-of-python-web-servers
uwsgi comes bundles with Nginx? First I've heard of that. Very interesting.
Do you have any easy tutorials on getting Nginx + uwsgi set up?