Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Deployment Anti-Patterns

hynek.me

Post reply on HN
Print view View original

21–30 of 125 posts

Re: Python Deployment Anti-Patterns

#21

Post by bickfordb »

Long-time Python user here.

For a lot of my projects I write a shell script which builds all of the application dependencies (including services) into a project directory and run them all from there.

It takes a little bit of work to get going --- especially when building a new service for the first time --- but I like that it side-steps language-specific packaging tools (particularly the half-baked Python ones) and lets me pin an applications dependencies and port to various environments (develop on Mac, deploy on Unix) almost exactly. Integrating with Puppet/Chef is just a matter of breaking up the shell script into pieces.

Re: Python Deployment Anti-Patterns

#22

Post by themanr »

hynek wrote: post #3
Pewpewarrows wrote: post #2

Regarding "Don't use ancient system Python versions" and "Use virtual environments", you can knock out two birds with one stone by just using pythonbrew. It also saves you the hassle of rolling your own deb/rpm if a package doesn't happen to exist. Also, Chef/Puppet aren't "alternatives" to something like Fabric. Use the former for server provisioning, and use the latter for actually kicking off the deployment proces…

> Those tasks are better suited for Fabric's imperative design. Yes they are, but IMHO not on the target servers. I use Fabric to build DEBs that get deployed by Puppet. I prefer to have no build tools on target servers, YMMV.

Fabric isn't normally deployed on target servers though - it connects to them through SSH.

Re: Python Deployment Anti-Patterns

#23

Post by seliopou »

jordanb wrote: post #20

This guy seems to be of the opinion that the software should be completely isolated from the deployment operating system. I know that's a common view and wrapping as much of the site as possible up in a virtualenv certainly has a lot of advantages. But ultimately, your software is going to have to interact with the OS, at some level, otherwise, why do you even have an OS? So the question is: where do you draw the lin…

Point of fact, he does give his opinion on the postgresql issue: "We also have to compile our own Apaches and MySQLs because we need that fine-grained control."

But that's besides the point. I don't think he's arguing that software should be completely isolated form the deployment operating system. That would be absurd, since, as you pointed out, software has to interact with the OS at some level, i.e., to manage system resources. Just because the OS ships with a bunch of packages with specific versions doesn't mean you have to use them. And what I think the OP is saying is that to make your applications portable and easily deployable, you shouldn't.

Re: Python Deployment Anti-Patterns

#24

Post by jmlane »

postfuturist wrote: post #11

We do this, but also add a couple layers of safety between us and PyPI: 1. Run your own secure, local pypi clone with exact source versions of the packages you use. 2. The packages for production are built into RPMs from the local pypi. PyPI is great for discovery, getting things running quickly, and testing new versions, but you never want to rely on it, even for development.

I'd really enjoy reading about how to setup a PyPI mirror like the one you use in your development/deployment workflow. It seems like a really good idea, considering I've had problems with PyPI at really inconvenient times in the past.

Re: Python Deployment Anti-Patterns

#25

Post by shuzchen »

smacktoward wrote: post #18
heretohelp wrote: post #15

Earlier quoted context omitted.

They rarely have a modern enough package to be worthwhile. When you control the package pipeline, you have the ability to dictate how modern or conservative your software is. The problem is that most programming languages (especially Python and Ruby) are ecosystems unto themselves and often move at a much faster pace than any stable distro (or LTS) could keep up with. That's why we have gems and pip. Cf. Ubuntu LTS M…

Perhaps, but that's a completely different argument than the one TFA is making. Besides, in the Ubuntu case anyway, LTS is supposed to be old. The whole point of LTS releases is to let slow-moving institutions/enterprises sit on ancient packages for 3-5 years without having to worry about backporting security updates. If you want recent versions of packages LTS is precisely the wrong place for you to be.

I would disagree with this. LTS (I'm referring to Ubuntu LTS as well as RHEL) is still a good fit because there are 9001 other packages your system relies on for day-to-day operations, and I'd rather mess with those as least often as possible. If you don't use a LTS then you'll have to do whole system upgrades every few years - or else you stop getting security backports. I'd much rather micromanage the dozen highly visible packages that are immediately relevant to my app.

Also, even if you're not on an LTS, doesn't mean you have the latest/greatest available. The python community moves at it's own pace, so there's still a chance that you'll be stuck with the just-before-latest-stable version.

Re: Python Deployment Anti-Patterns

#26

Post by jordanb »

seliopou wrote: post #23
jordanb wrote: post #20

This guy seems to be of the opinion that the software should be completely isolated from the deployment operating system. I know that's a common view and wrapping as much of the site as possible up in a virtualenv certainly has a lot of advantages. But ultimately, your software is going to have to interact with the OS, at some level, otherwise, why do you even have an OS? So the question is: where do you draw the lin…

Point of fact, he does give his opinion on the postgresql issue: "We also have to compile our own Apaches and MySQLs because we need that fine-grained control." But that's besides the point. I don't think he's arguing that software should be completely isolated form the deployment operating system. That would be absurd, since, as you pointed out, software has to interact with the OS at some level, i.e., to manage sys…

Good catch. Sorry I missed that. I suppose I've forgotten that a lot of people still use MySQL, even for python. :)

The question "why even have an OS?" was a rhetorical one, meant to illustrate the fact that the argument here is over where one draws the line, between the dependencies you maintain yourself and those you outsource to your OS vendor. Unless you're deploying on Linux From Scratch then you are outsourcing to the vendor at some level, so the only question is, where is that level?

That this guy maintains even his own web and database servers means wants/needs to control things at a much deeper level than is typical. And I think that's beautiful if it works for him. My point of disagreement is his use of the word "antipattern" to suggest that any other choice is wrong.

Re: Python Deployment Anti-Patterns

#27

Post by lifeisstillgood »

Ixiaus wrote: post #8
djtriptych wrote: post #4

As a python dev who deploys a lot of software, I found this article to be wonderfully helpful and informative, and a good reflection of current best practices. Summary of the deployment tools mentioned: - Manage remote daemons with supervisord http://supervisord.org/ - Manage python packages with pip (and use `pip freeze`) http://pypi.python.org/pypi/pip http://www.pip-installer.org/en/latest/requirements.html - Mana…

Yes to all of this and I'll throw in my two cents on our web server setup which is nginx + uwsgi (which has served remarkably well).

+1 to uWSGI and nginx. Now wsgi is built into nginx distribution, and uwsgi is gaining areputation for rock solid performance it's a nobrainer.

Re: Python Deployment Anti-Patterns

#28

Post by viraptor »

hynek wrote: post #17
smacktoward wrote: post #14

I don't get the negativity on using your distro's packages, at least from the staying-stable perspective. Any decent package manager should let you pin/hold critical packages on a particular version, so if "the next Ubuntu ships with a different SQLAlchemy by default" you just hold the SQLAlchemy package at the version you want and then ignore it until you're ready to make that move. 99% percent of the time when I he…

The worst thing about them is the fact that they are installed into global site-packages that you shouldn’t use for any serious coding. And yes, they are mostly outdated too.

> "into global site-packages that you shouldn’t use for any serious coding"

Any specific reason for that? I find it quite good and have quite large deployments using .debs only with packages in global location. (tens of packages produced locally - either updated or unavailable dependencies and the service itself) Any direct dependency is handled by package pinning and no update goes into production untested, so the whole "new sqlalchemy suddenly appears" issue does not exist. As long as people don't break API versioning in silly ways, what's the problem with this?

The only version-related issue I remember was when someone thought it would be nice to install something through pip, instead of via package. (went to /usr/local)

Re: Python Deployment Anti-Patterns

#29

Post by hynek »

jordanb wrote: post #26
seliopou wrote: post #23

Earlier quoted context omitted.

Point of fact, he does give his opinion on the postgresql issue: "We also have to compile our own Apaches and MySQLs because we need that fine-grained control." But that's besides the point. I don't think he's arguing that software should be completely isolated form the deployment operating system. That would be absurd, since, as you pointed out, software has to interact with the OS at some level, i.e., to manage sys…

Good catch. Sorry I missed that. I suppose I've forgotten that a lot of people still use MySQL, even for python. :) The question "why even have an OS?" was a rhetorical one, meant to illustrate the fact that the argument here is over where one draws the line, between the dependencies you maintain yourself and those you outsource to your OS vendor. Unless you're deploying on Linux From Scratch then you are outsourcing…

First of all I’m afraid the point about Apache & MySQL is misleading and I’ll have to clarify it: We don’t use neither for internal projects. It’s all PostgreSQL. :)

They’re both for our customers as we sell traditional web hosting with a LAMP stack (and yes, we have to compile PHP ourselves too, so we can offer different flavors).

I never said you _have_ to compile everything yourself. In case of Python I said you shouldn’t inflict the pain of programming Python 2.4 just because you’re on RHEL/CentOS 5 or earlier.

System packages have several problems I outlined too. Most important: you add a middleman to the release pipeline, no virtualenv and possible dependency conflicts: I have apps that depend both SQLAlchemy 0.6 and 0.7 – they couldn’t run on the same server.

But OTOH we use stock packages as far as possible, because it’s less work. Eg. there’s no reason to compile an own Python on Oneiric and later. Same goes for PostgreSQL which is uptodate.

It’s all about freeing yourself from constrains in points that matter, not adopting a new religion.

Re: Python Deployment Anti-Patterns

#30

Post by [deleted] »

Pewpewarrows wrote: post #2

Regarding "Don't use ancient system Python versions" and "Use virtual environments", you can knock out two birds with one stone by just using pythonbrew. It also saves you the hassle of rolling your own deb/rpm if a package doesn't happen to exist. Also, Chef/Puppet aren't "alternatives" to something like Fabric. Use the former for server provisioning, and use the latter for actually kicking off the deployment proces…

[deleted]
Post reply on HN