Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Deployment Anti-Patterns

hynek.me

Post reply on HN
Print view View original

41–50 of 125 posts

Re: Python Deployment Anti-Patterns

#41

Post by mdehaan »

djtriptych wrote: post #4

As a python dev who deploys a lot of software, I found this article to be wonderfully helpful and informative, and a good reflection of current best practices. Summary of the deployment tools mentioned: - Manage remote daemons with supervisord http://supervisord.org/ - Manage python packages with pip (and use `pip freeze`) http://pypi.python.org/pypi/pip http://www.pip-installer.org/en/latest/requirements.html - Mana…

Chef requires Ruby programming. Puppet doesn't, but the core is obviously Ruby, and you extend it using Ruby.

You may possibly like my new project:

http://ansible.github.com

The core is Python but you can write modules in any language.

Re: Python Deployment Anti-Patterns

#42

Post by hynek »

viraptor wrote: post #36
hynek wrote: post #32

Earlier quoted context omitted.

What is the benefit of using global packages? If you pin them, what do you gain? virtualenv give you complete control over the runtime environment of every single application. Stuff like what you wrote can’t really happen. And BTW, there’s more stuff that can happen than breaking APIs: new bugs that happen only on your system or even better: your code worked only _because_ of a bug. :)

For me the benefit is a much easier security audit/update. Unless you have dependencies packaged by yourself, the only thing you need to monitor is what comes into security-updates of your repository. If there's anything new, do a test deployment, do what you do to verify it's correct and change the pin in production. Honestly: how many people installing software through virtualenv are registered to security mailing…

But if you _pin_ your packages, how are you updating? You have to monitor it anyway.

High profile projects like simplejson, Django or Pyramid and their deps won’t be missed and the really obscure ones will never make it into the repositories anyway.

Re: Python Deployment Anti-Patterns

#43

Post by hynek »

themanr wrote: post #35
hynek wrote: post #33

Earlier quoted context omitted.

That’s not what I meant: If you build your virtualenv on the target server, you need build tools like GCC or development files like libpq-dev. I prefer to have as few stuff on servers as possible.

Thanks - that does sound like a big advantage of the deb / rpm route. Plus you don't really want your app servers spending their CPU time compiling.

That’s a plus too. An “aptitude dist-upgrade” is really fast.

Re: Python Deployment Anti-Patterns

#44

Post by raverbashing »

jordanb wrote: post #20

This guy seems to be of the opinion that the software should be completely isolated from the deployment operating system. I know that's a common view and wrapping as much of the site as possible up in a virtualenv certainly has a lot of advantages. But ultimately, your software is going to have to interact with the OS, at some level, otherwise, why do you even have an OS? So the question is: where do you draw the lin…

I think there's an exageration as well, today the trend seems to be "use nothing by the distro"

Ok, sure, MongoDB still changes a lot between versions, in this case you should use the latest version.

But stop there. Especially if you're paying for support (like RHEL)

There should be a good reason for you to compile Apache / MySQL / PostgreSQL / Python. Otherwise, use the distro version. One (common) exception would be "we need Python 2.7 but this ships only 2.6"

Most of the "just download and compile" have no idea of the work that goes behind Linux distributions to ship these packages.

Yes, I'm sure you're going to read all security advisories and recompile all your stack every X days instead of running apt/yum upgrade

Re: Python Deployment Anti-Patterns

#45

Post by mbreese »

viraptor wrote: post #40
j_baker wrote: post #37

Earlier quoted context omitted.

Having a global python installation where packages are constantly installed, uninstalled, and updated is the path to madness. If something goes wrong, what can you do? You can't wipe out the system python. You can wipe out a virtualenv though.

What do you mean by goes wrong? Either some package is installed or not. For me, chef manages which ones are. If something is really FUBAR, then wiping is exactly the path I'd take - or more accurately, take that server down for analysis of how exactly it got into that state (so we won't do that again) and bring a clean one up.

Mainly it's about incompatibilities. What if you have two apps that require different versions of a library? If you've installed it in site-packages, then you have little recourse. By separating them out with virtualenv the two apps will work just fine.

Re: Python Deployment Anti-Patterns

#46

Post by viraptor »

hynek wrote: post #42
viraptor wrote: post #36

Earlier quoted context omitted.

For me the benefit is a much easier security audit/update. Unless you have dependencies packaged by yourself, the only thing you need to monitor is what comes into security-updates of your repository. If there's anything new, do a test deployment, do what you do to verify it's correct and change the pin in production. Honestly: how many people installing software through virtualenv are registered to security mailing…

But if you _pin_ your packages, how are you updating? You have to monitor it anyway. High profile projects like simplejson, Django or Pyramid and their deps won’t be missed and the really obscure ones will never make it into the repositories anyway.

Of course. I'm just saying that in case you're pinning only the default upstream versions from your distro so that they don't change, it's easier to automatically report on which packages have a new version in {distro}-security repository. Then retest and change the pin.

The same can be achieved by subscribing to CVEs... but you have to remember to filter the ones you use. Of course that's not a huge difference, so if someone prefers the second way, there's nothing wrong with it ;)

Re: Python Deployment Anti-Patterns

#47

Post by viraptor »

mbreese wrote: post #45
viraptor wrote: post #40

Earlier quoted context omitted.

What do you mean by goes wrong? Either some package is installed or not. For me, chef manages which ones are. If something is really FUBAR, then wiping is exactly the path I'd take - or more accurately, take that server down for analysis of how exactly it got into that state (so we won't do that again) and bring a clean one up.

Mainly it's about incompatibilities. What if you have two apps that require different versions of a library? If you've installed it in site-packages, then you have little recourse. By separating them out with virtualenv the two apps will work just fine.

Fortunately I'm in a one-service-one-server environment, so I may be biased here ;)

Re: Python Deployment Anti-Patterns

#48

Post by hynek »

raverbashing wrote: post #44
jordanb wrote: post #20

This guy seems to be of the opinion that the software should be completely isolated from the deployment operating system. I know that's a common view and wrapping as much of the site as possible up in a virtualenv certainly has a lot of advantages. But ultimately, your software is going to have to interact with the OS, at some level, otherwise, why do you even have an OS? So the question is: where do you draw the lin…

I think there's an exageration as well, today the trend seems to be "use nothing by the distro" Ok, sure, MongoDB still changes a lot between versions, in this case you should use the latest version. But stop there. Especially if you're paying for support (like RHEL) There should be a good reason for you to compile Apache / MySQL / PostgreSQL / Python. Otherwise, use the distro version. One (common) exception would b…

Yes you should, but if you have them: do it. That’s what the article says. ;) Please don’t push it in a wrong direction.

What I actually wrote is: because we’re a LAMP web hoster, we compile MySQL+PHP+Apache ourself. And because we’re a Python shop, we don’t let Ubuntu/Red Hat dictate which Python version we use.

Re: Python Deployment Anti-Patterns

#49

Post by hynek »

eclark wrote: post #39
afhof wrote: post #34

Using tmux is a Python daemon antipattern? And then "there's so much wrong about this approach" that he doesn't bother explaining why? Isn't that why we are reading the article: because we want to know why? If the author is trying to convince people to change their habits, he is doing a crummy job. He comes across as elitist and "if you don't do it my way you're wrong".

He's talking about creating /etc/init.d scripts that do something like: screen python manage.py gunicorn & That is an anti-pattern. Granted he didn't really explain it so well.

Honestly I didn’t think it would be necessary. :-/

Your example is btw the advanced version. Many people just ssh on the host, fire up screen and start their server inside.

Re: Python Deployment Anti-Patterns

#50

Post by hynek »

afhof wrote: post #34

Using tmux is a Python daemon antipattern? And then "there's so much wrong about this approach" that he doesn't bother explaining why? Isn't that why we are reading the article: because we want to know why? If the author is trying to convince people to change their habits, he is doing a crummy job. He comes across as elitist and "if you don't do it my way you're wrong".

I’m sorry if that felt to you like that. I’ll consider rewording it.

update I added some more context. I would never spit on my beloved tmux. :)

Post reply on HN