Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Deployment Anti-Patterns

hynek.me

Post reply on HN
Print view View original

31–40 of 125 posts

Re: Python Deployment Anti-Patterns

#31

Post by viraptor »

jordanb wrote: post #20

This guy seems to be of the opinion that the software should be completely isolated from the deployment operating system. I know that's a common view and wrapping as much of the site as possible up in a virtualenv certainly has a lot of advantages. But ultimately, your software is going to have to interact with the OS, at some level, otherwise, why do you even have an OS? So the question is: where do you draw the lin…

If you use puppet/chef for the whole stack, you gain the ability of just starting a new machine and in a couple of minutes having it configured in the exact same way as all the others.

With fabric and similar systems it's a bit harder. Basically you'd have to write your scripts exactly the same way you'd write a puppet/chef recipe: "make sure this is configured that way, make sure that is installed", etc. (or do migration steps) It's very different from fabric's "do this, do that" approach. Unless you run fabric on every single host after you make every change, some of your infrastructure will be lagging behind.

For example, what do you do when you create a new server, or do an upgrade that involves different dependencies? Run a fabric script that migrates from state X to Y? What happens to new machines then? How do you make sure they're in the same state?

I found chef a very good solution even if I have a single server to manage. No need to think about how it was configured before. Migrating to another provider? Just migrate the data, point the server at chef, done.

Re: Python Deployment Anti-Patterns

#32

Post by hynek »

viraptor wrote: post #28
hynek wrote: post #17

Earlier quoted context omitted.

The worst thing about them is the fact that they are installed into global site-packages that you shouldn’t use for any serious coding. And yes, they are mostly outdated too.

> "into global site-packages that you shouldn’t use for any serious coding" Any specific reason for that? I find it quite good and have quite large deployments using .debs only with packages in global location. (tens of packages produced locally - either updated or unavailable dependencies and the service itself) Any direct dependency is handled by package pinning and no update goes into production untested, so the w…

What is the benefit of using global packages? If you pin them, what do you gain? virtualenv give you complete control over the runtime environment of every single application. Stuff like what you wrote can’t really happen.

And BTW, there’s more stuff that can happen than breaking APIs: new bugs that happen only on your system or even better: your code worked only _because_ of a bug. :)

Re: Python Deployment Anti-Patterns

#33

Post by hynek »

themanr wrote: post #22
hynek wrote: post #3

Earlier quoted context omitted.

> Those tasks are better suited for Fabric's imperative design. Yes they are, but IMHO not on the target servers. I use Fabric to build DEBs that get deployed by Puppet. I prefer to have no build tools on target servers, YMMV.

Fabric isn't normally deployed on target servers though - it connects to them through SSH.

That’s not what I meant: If you build your virtualenv on the target server, you need build tools like GCC or development files like libpq-dev. I prefer to have as few stuff on servers as possible.

Re: Python Deployment Anti-Patterns

#34

Post by afhof »

Using tmux is a Python daemon antipattern? And then "there's so much wrong about this approach" that he doesn't bother explaining why? Isn't that why we are reading the article: because we want to know why?

If the author is trying to convince people to change their habits, he is doing a crummy job. He comes across as elitist and "if you don't do it my way you're wrong".

Re: Python Deployment Anti-Patterns

#35

Post by themanr »

hynek wrote: post #33
themanr wrote: post #22

Earlier quoted context omitted.

Fabric isn't normally deployed on target servers though - it connects to them through SSH.

That’s not what I meant: If you build your virtualenv on the target server, you need build tools like GCC or development files like libpq-dev. I prefer to have as few stuff on servers as possible.

Thanks - that does sound like a big advantage of the deb / rpm route. Plus you don't really want your app servers spending their CPU time compiling.

Re: Python Deployment Anti-Patterns

#36

Post by viraptor »

hynek wrote: post #32
viraptor wrote: post #28

Earlier quoted context omitted.

> "into global site-packages that you shouldn’t use for any serious coding" Any specific reason for that? I find it quite good and have quite large deployments using .debs only with packages in global location. (tens of packages produced locally - either updated or unavailable dependencies and the service itself) Any direct dependency is handled by package pinning and no update goes into production untested, so the w…

What is the benefit of using global packages? If you pin them, what do you gain? virtualenv give you complete control over the runtime environment of every single application. Stuff like what you wrote can’t really happen. And BTW, there’s more stuff that can happen than breaking APIs: new bugs that happen only on your system or even better: your code worked only _because_ of a bug. :)

For me the benefit is a much easier security audit/update. Unless you have dependencies packaged by yourself, the only thing you need to monitor is what comes into security-updates of your repository. If there's anything new, do a test deployment, do what you do to verify it's correct and change the pin in production.

Honestly: how many people installing software through virtualenv are registered to security mailing lists for each of their packages (and their dependencies down to things like simplejson)?

Re: Python Deployment Anti-Patterns

#37

Post by j_baker »

viraptor wrote: post #28
hynek wrote: post #17

Earlier quoted context omitted.

The worst thing about them is the fact that they are installed into global site-packages that you shouldn’t use for any serious coding. And yes, they are mostly outdated too.

> "into global site-packages that you shouldn’t use for any serious coding" Any specific reason for that? I find it quite good and have quite large deployments using .debs only with packages in global location. (tens of packages produced locally - either updated or unavailable dependencies and the service itself) Any direct dependency is handled by package pinning and no update goes into production untested, so the w…

Having a global python installation where packages are constantly installed, uninstalled, and updated is the path to madness. If something goes wrong, what can you do? You can't wipe out the system python. You can wipe out a virtualenv though.

Re: Python Deployment Anti-Patterns

#38

Post by j_baker »

smacktoward wrote: post #14

I don't get the negativity on using your distro's packages, at least from the staying-stable perspective. Any decent package manager should let you pin/hold critical packages on a particular version, so if "the next Ubuntu ships with a different SQLAlchemy by default" you just hold the SQLAlchemy package at the version you want and then ignore it until you're ready to make that move. 99% percent of the time when I he…

It's usually best to leave these packages for the system python to run pieces of the system. There's one exception that I usually make for this rule: packages with complex C dependencies. NumPy is the first package that comes to mind. I usually prefer to just use the package manager to deal with that.

Re: Python Deployment Anti-Patterns

#39

Post by eclark »

afhof wrote: post #34

Using tmux is a Python daemon antipattern? And then "there's so much wrong about this approach" that he doesn't bother explaining why? Isn't that why we are reading the article: because we want to know why? If the author is trying to convince people to change their habits, he is doing a crummy job. He comes across as elitist and "if you don't do it my way you're wrong".

He's talking about creating /etc/init.d scripts that do something like:

screen python manage.py gunicorn &

That is an anti-pattern. Granted he didn't really explain it so well.

Re: Python Deployment Anti-Patterns

#40

Post by viraptor »

j_baker wrote: post #37
viraptor wrote: post #28

Earlier quoted context omitted.

> "into global site-packages that you shouldn’t use for any serious coding" Any specific reason for that? I find it quite good and have quite large deployments using .debs only with packages in global location. (tens of packages produced locally - either updated or unavailable dependencies and the service itself) Any direct dependency is handled by package pinning and no update goes into production untested, so the w…

Having a global python installation where packages are constantly installed, uninstalled, and updated is the path to madness. If something goes wrong, what can you do? You can't wipe out the system python. You can wipe out a virtualenv though.

What do you mean by goes wrong? Either some package is installed or not. For me, chef manages which ones are. If something is really FUBAR, then wiping is exactly the path I'd take - or more accurately, take that server down for analysis of how exactly it got into that state (so we won't do that again) and bring a clean one up.
Post reply on HN