Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Deployment Anti-Patterns

hynek.me

Post reply on HN
Print view View original

81–90 of 125 posts

Re: Python Deployment Anti-Patterns

#81

Post by mixmastamyk »

mdehaan wrote: post #41
djtriptych wrote: post #4

As a python dev who deploys a lot of software, I found this article to be wonderfully helpful and informative, and a good reflection of current best practices. Summary of the deployment tools mentioned: - Manage remote daemons with supervisord http://supervisord.org/ - Manage python packages with pip (and use `pip freeze`) http://pypi.python.org/pypi/pip http://www.pip-installer.org/en/latest/requirements.html - Mana…

Chef requires Ruby programming. Puppet doesn't, but the core is obviously Ruby, and you extend it using Ruby. You may possibly like my new project: http://ansible.github.com The core is Python but you can write modules in any language.

Cool, I'm also investigating the python-based salt stack: http://saltstack.org/ for this purpose but it seems a bit heavy just starting out. Gonna try ansible next.

Re: Python Deployment Anti-Patterns

#82

Post by mixmastamyk »

Ixiaus wrote: post #8
djtriptych wrote: post #4

As a python dev who deploys a lot of software, I found this article to be wonderfully helpful and informative, and a good reflection of current best practices. Summary of the deployment tools mentioned: - Manage remote daemons with supervisord http://supervisord.org/ - Manage python packages with pip (and use `pip freeze`) http://pypi.python.org/pypi/pip http://www.pip-installer.org/en/latest/requirements.html - Mana…

Yes to all of this and I'll throw in my two cents on our web server setup which is nginx + uwsgi (which has served remarkably well).

I chose nginx and gevent after reading a blog post about py server performance. Anyone have any experience with these two?

Re: Python Deployment Anti-Patterns

#83

Post by hynek »

njharman wrote: post #69

Virtualenv is a half solution and a hack. Use vagrant and VMs. There's a whole sea of libs and software that isn't "versioned" by pip/virtualenv. supervisord is the wrong solution. It answers the wrong question (is the process running). It's worse than useless in that it has given false positives. The right question is (is the process responding correctly). Use monit or something else that actually does what's needed…

Vagrant ist nice for testing but I’m not going to deploy services to VirtualBox (we use kvm for that). Of course there are dependencies outside of the Python ecosystem but their influence proved to be negligible till now. YMMV.

supervisor is not a an alternative to monitoring, I never claimed that. But that’s a whole different story.

Re: Python Deployment Anti-Patterns

#84

Post by mrweasel »

antihero wrote: post #78
njharman wrote: post #69

Virtualenv is a half solution and a hack. Use vagrant and VMs. There's a whole sea of libs and software that isn't "versioned" by pip/virtualenv. supervisord is the wrong solution. It answers the wrong question (is the process running). It's worse than useless in that it has given false positives. The right question is (is the process responding correctly). Use monit or something else that actually does what's needed…

Why would you want to virtualise an entire system when your dependencies are restricted to a bunch of Python modules?

Personally I would just go for Virtualenv, but you'd be surprised if you know people do with virtualization. I've mostly seen it in Windows shops, but it's by no means limited to that. A common "anti-pattern" I've seen is to build a vmware image, or set of images, that run your "environment". Developers then spin up copies and request changes that they need be made to the "master copy".

When it's time to test and deploy, the code is deployed to a vmware image, which is then sent to the testers. When everything checks out, the code is once again deploy to a new copy of the image, which is then promoted to production.

On argument we've seen is that it makes it easy to do a rollback of the system, using vmware snapshots. It might be a sensible idea in some cases, I just thing it's a bit weird and some overhead.

The worst use case of this I've seen is a Telco, where you needed to spin up as much 12 vms, depending one what you and your teams was working on. But then again that's the same company that read the Perforce guidelines on recommended setup and still decided to just pill all the projects in to one repository.

Re: Python Deployment Anti-Patterns

#85

Post by hynek »

cdavid wrote: post #68

I am glad this is working for the OP, but pushing virtualenv and "self-contained" apps as the one solution is a diservice to the community. There are valid reasons to rely on your OS, assuming you have an homogenous deployment target (same OS, maybe different versions): - lots of people argue for virtualenv because some versions may be incompatible. The problem here is the lack of backward compatibility of packages,…

> pushing virtualenv and "self-contained" apps as the one solution is a diservice to the community.

Wow. :(

> There are valid reasons to rely on your OS, assuming you have an homogenous deployment target (same OS, maybe different versions):

I’d love to hear them.

> lots of people argue for virtualenv because some versions may be incompatible. The problem here is the lack of backward compatibility of packages, and frankly, if you need to rely on packages which change API willingnily betwen e.g. 1.5 and 1.6, or if each of your service depends on a different version of some library, you have bigger problems anyway.

Well, you said there are possibilities of problems but that they shouldn’t matter in an ideal world. Maybe you’re okay to take the chances but I’m not. Every code I deploy has been tested rigorously against a certain set of versions and that is the only combination of dependencies I’m willing to consider “working”. UnitTests with different dependencies are just as worthless like integration/functional tests against sqllite instead of the same DB type as in production.

There’s even the possibility that your code works because of a bug and when that one gets fixed, you app goes south because of some weird side-effect.

> any sufficiently complex deployment will depend on things that are not python, at which point you need a solution that integrates multiple languages. That is, you re-creating what a distribution is all about.

I’m not sure if I understand what you mean, but yes if you want to use certain features outside the Python ecosystem, you’ll have to buckle up and package them yourself too. “We can’t do that, package XYZ is missing/too old.” isn’t really a good excuse to not do something that is important/good for your business. And that’s one of the main points of the article.

> virtualenv relies on sources, so if some of your dependences are in C, every deploy means compilation

That’s wrong if you go the way described: The virtualenv is packaged with the code. Build tools don’t belong on production servers.

> I still have no idea how security is handled when you put everything in virtualenv

Just as everywhere else. If you think it’s okay to tell customers that their data has been hacked because debian was to slow to issue a fix, be my guest. We can’t afford that. What happens on my servers security wise is _my_ responsibility and using ancient versions of Python libraries just to be able to blame others for FUBARs is not a solution in my book.

Re: Python Deployment Anti-Patterns

#86

Post by hynek »

grncdr wrote: post #65
hynek wrote: post #53

Earlier quoted context omitted.

Fabric is an invaluable deployment tool, but you have to know its boundaries for what it was intended. With Fabric you tell what to _do_ and with Puppet/Chef you define what the result should _look_ like. You define how a server should look like and it can make sure its true for 1000 servers. Or 10000. It’s not about Fabric vs. Puppet, but how to use both in the most efficient way.

I find the line between when to use fabric and when to use puppet/chef is needing the permissions of a more privileged user. I prefer to deploy applications in the home directory of a dedicated user account with minimal privileges, and use fabric for installing updates and running application tasks. OTOH, I'd prefer to be using puppet for creating the user accounts, managing the installation and configuration of Post…

That’s exactly how we do it too. The only difference is that we don’t install the app using Fabric. Instead, puppet installs the deb with the app into the directory.

Maybe I should add a “running apps as root” anti-pattern, but this its 2012 after all, everyone should know that, right? :-/

Re: Python Deployment Anti-Patterns

#88

Post by tartley »

antihero wrote: post #77
hynek wrote: post #53

Earlier quoted context omitted.

Fabric is an invaluable deployment tool, but you have to know its boundaries for what it was intended. With Fabric you tell what to _do_ and with Puppet/Chef you define what the result should _look_ like. You define how a server should look like and it can make sure its true for 1000 servers. Or 10000. It’s not about Fabric vs. Puppet, but how to use both in the most efficient way.

It would be fantastic if someone were to write a tutorial how to deploy, for instance, a simple Flask+MySQL or Django+PostgreSQL application using these tools. I'm at a loss as to where to start.

http://ontwik.com/python/django-deployment-workshop-by-jacob...

Re: Python Deployment Anti-Patterns

#89

Post by davvid »

ch0wn wrote: post #5

> The trick is to build a debian package (but it can be done using RPMs just as well) with the application and the whole virtualenv inside. I would love to read an article describing some best practices for doing that. I tried it once and found it extremely difficult, reverting to a git checkout + virtualenv kind of deployment.

Check out git's "make rpm" target.

https://github.com/gitster/git/blob/master/Makefile

Hosting your own apt/yum repo is pretty simple.

Does anyone have an example of a similar "make deb" target they could share?

I've heard of git-dpm and git-buildpackage but haven't used them extensively myself. They're the debian git packaging tools.

http://wiki.debian.org/PackagingWithGit

Re: Python Deployment Anti-Patterns

#90

Post by hynek »

davvid wrote: post #89
ch0wn wrote: post #5

> The trick is to build a debian package (but it can be done using RPMs just as well) with the application and the whole virtualenv inside. I would love to read an article describing some best practices for doing that. I tried it once and found it extremely difficult, reverting to a git checkout + virtualenv kind of deployment.

Check out git's "make rpm" target. https://github.com/gitster/git/blob/master/Makefile Hosting your own apt/yum repo is pretty simple. Does anyone have an example of a similar "make deb" target they could share? I've heard of git-dpm and git-buildpackage but haven't used them extensively myself. They're the debian git packaging tools. http://wiki.debian.org/PackagingWithGit

Have a look at fpm: https://github.com/jordansissel/fpm

More details to come.

Post reply on HN