Python Deployment Anti-Patterns

Yes, but unless rpm/deb works with softlinks, it is not atomic right? Do you do anything before/after dist-upgrade or is it included in the pre/post install script?

For example, for our deployment, we rely on softlinks and uwsgi robust reload behavior to avoid losing requests. I've seen many devops who were using hg update/git update as a way to "deploy" (arg!), but I'm not sure about the behavior of deb/rpm.

> I’m not sure if I understand what you mean, but yes if you want to use certain features outside the Python ecosystem, you’ll have to buckle up and package them yourself too.

I think his point is that pip can't do this, but learning to actually work with your distro's packaging system properly results in a more powerful and easier to redistribute project.

> Just as everywhere else. If you think it’s okay to tell customers that their data has been hacked because debian was to slow to issue a fix, be my guest.

Your packaging methodology is not what gives you security there. It's that you noticed a vulnerability and deployed a fix. The method of deployment is irrelevant. Your point is that knowing you have a security issue and waiting for upstream to get around to fixing it isn't always acceptable. That goes for everything.

If you know how to roll .debs it's just as easy to patch and release a fixed version of a library. (Or even install the pip version earlier in your sys.path...)

My two cents: I am a developer + ops person and deploy Python apps all the time. Typically they are Django and Tornado services. On top we also have a lot of daemons and a ton of library code.

I agree with the OP on most points but do not on a few. First DO use packages that come with the OS. The OP says that you should not have the distro maintainers dictating what you use. I say, use what is widely available. It takes the headache out of a lot of your deployments. If you are looking for a library that converts foo to bar look in your distro's repos before going on GitHub. Your sysadmin will thank you.

Second, DO NOT use virtualenv. It fixes the symptoms (Python's packaging system has many shortcomings such as inability to uninstall recursively, poor dependency management, lack of pre and post install scripts, etc.), but not the problem. Instead, use distro-appropriate packages. Integrate your app into the system. This way you will never end up running a daemon inside a screen session, etc. You also get the ability to very nicely manage dependencies and a clean separation between code and configuration.

Lastly, DO use apache + mod_wsgi. It is fast, stable, widely supported and well tested. If apache feels like a ball of mud, take the time to understand how to cut it down to a minimum and configure it properly.

When it comes to infrastructure, making boring choices leads to predictable performance and less headaches more often than not (at least in my experience).

> First DO use packages that come with the OS. The OP says that you should not have the distro maintainers dictating what you use. I say, use what is widely available. It takes the headache out of a lot of your deployments. If you are looking for a library that converts foo to bar look in your distro's repos before going on GitHub. Your sysadmin will thank you.

The sysadmin will have no part in the game if you use packaged virtualenvs. OTOH developer time is expensive. Do you really want to pay your developers to implement functionality that a more recent version of a package has already implemented? A good example is IPv6 support in Twisted. It’s getting implemented right now but I guess (and hope) that I’ll need it sooner than it lands in major distros (please no “lol ipv6” here, it’s just an example and the support is growing).

> Second, DO NOT use virtualenv. It fixes the symptoms (Python's packaging system has many shortcomings such as inability to uninstall recursively, poor dependency management, lack of pre and post install scripts, etc.), but not the problem. Instead, use distro-appropriate packages.

I’m not sure what your problem is, but mine is that I don’t want to develop against a moving target and need to run apps with contradicting dependencies on the same host.

That’s how I started using virtualenv years ago BTW, I’m not talking ivory tower here.

> Integrate your app into the system.

Yes. And I prefer supervisor for that. If your prefer rc.d scripts, be my guest.

> You also get the ability to very nicely manage dependencies and a clean separation between code and configuration.

I don’t get this one TBH.

> Lastly, DO use apache + mod_wsgi. It is fast, stable, widely supported and well tested.

And nginx + uwsgi/gunicorn aren’t?

> If apache feels like a ball of mud, take the time to understand how to cut it down to a minimum and configure it properly.

I know Apache pretty well, because we’re running thousands of customers on them. I’ve already written modules for it and been more than once in it‘s guts. And my impression is not a good one.

My point was to look around before you settle. If you think Apache is da best, knock yourself out. However stuff I see daily on IRC lets me think that it isn’t very unproblematic.

I’m not going to start a “vi vs. emacs”-style holy war here. That’s why I wrote “shop around before you settle” and not ”don’t ever use Apache”.

> When it comes to infrastructure, making boring choices leads to predictable performance and less headaches more often than not (at least in my experience).

Absolutely. nginx is way past the “new and hacky” state though.

There is also cuisine, which brings chef-like functionality to fabric.

https://github.com/sebastien/cuisine