Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Deployment Anti-Patterns

hynek.me

Post reply on HN
Print view View original

111–120 of 125 posts

Re: Python Deployment Anti-Patterns

#111

Post by lifeisstillgood »

j_baker wrote: post #59
lifeisstillgood wrote: post #51

Regarding virtualenv, I have come to the conclusion that Linux containers are robust enough now (like freebsd jails say two or three years ago) that I don't need to virtualise just python - I can afford to have the whole server as a "virtualenv" - no need for that extra complexity just install into site packages. No conflicts because a whole instance is dedicated. Jails take this to the limit - one virtual machine, o…

Sorry, but I'm not sure I understand your logic. Using virtualenv adds extra complexity, but virtualizing the entire server doesn't? I mean, the only complexity using virtualenv adds is having to run the virtualenv process once. After that, you can still install to site-packages. You just have to install to a different site-packages directory. Besides that, it's worth pointing out that using a virtualenv is not a sec…

A late reply but I don't just want the python environment virtualised - if it's important enough that I should section off python, then there is a good chance I will want to consider the whole box as a single unit, python, firewall rules, database what ever. I tend to think the unit of abstraction should not be the python process, but the server. This is a little easier to grasp when you think of BSD jails where essentially you can choose to only run those processes that actually mTter - it's less virtualised and os than pick and mix an os.

Apologies for late reply - I guess I am straightening it out in my head more than telling anyone else.

Re: Python Deployment Anti-Patterns

#112

Post by shuzchen »

sliverstorm wrote: post #55
shuzchen wrote: post #25

Earlier quoted context omitted.

I would disagree with this. LTS (I'm referring to Ubuntu LTS as well as RHEL) is still a good fit because there are 9001 other packages your system relies on for day-to-day operations, and I'd rather mess with those as least often as possible. If you don't use a LTS then you'll have to do whole system upgrades every few years - or else you stop getting security backports. I'd much rather micromanage the dozen highly…

there's still a chance that you'll be stuck with the just-before-latest-stable version. Is that so bad? Does python really change that much from release to release?

Not entirely, but if your reason to using a non-LTS is to get the latest versions of your stack, then you've just made a really bad choice. As I've said, you're either giving up security updates or are left to test and upgrade your entire system every ~two years in exchange for no guarantee you're on the latest of anything. That tradeoff doesn't make sense no matter what way you look at it.

Re: Python Deployment Anti-Patterns

#113

Post by steviesteveo65 »

heretohelp wrote: post #66
sliverstorm wrote: post #55

Earlier quoted context omitted.

there's still a chance that you'll be stuck with the just-before-latest-stable version. Is that so bad? Does python really change that much from release to release?

You don't really want to be on anything earlier than 2.7 right now.

Well, depending on your program.

Re: Python Deployment Anti-Patterns

#114

Post by IgorPartola »

hynek wrote: post #104
IgorPartola wrote: post #103

Earlier quoted context omitted.

Sysadmin/Ops should have a part in what gets deployed. It is their job. My point is that if you have a choice between Python 2.6 that comes with your distro and Python 2.7 that doesn't, go with 2.6. The cost is minimal. Same for various libraries (PIL, NumPy, etc.) I currently have to deploy Django-Piston directly from a master branch on GitHub due to some terrible decisions made by developers who never gave deployme…

> I currently have to deploy Django-Piston directly from a master branch on GitHub due to some terrible decisions made by developers who never gave deployment a second thought. Avoid this. JFTR, there is a pretty good solution for this (because in the real world, you can’t always avoid that): a custom pypi server. You take a git version you know that works do a `python setup.py sdist` und push it to a private repo. W…

Nive tip about the private PyPI server. I might use it in the future if we end up in the same situation again.

Also, one more nice thing about using distro packages for your own deploys: if you ever need to mix Python with anything else it is a godsend. For example, at one point I had a network sniffer/userland packet forwarder that was written in C deployed alongside our Python daemons. Debian packages do not care what you put in them: C, Python, JavaScript or pure data.

Re: Python Deployment Anti-Patterns

#116

Post by joeshaw »

dignan wrote: post #105
alexatkeplar wrote: post #57

Earlier quoted context omitted.

Likewise +1 to Nginx + uwsgi. In the various performance benchmarks* uwsgi beats gunicorn, and uwsgi comes bundled with Nginx now, so there's really no reason to bother with setting up gunicorn any more. * http://www.peterbe.com/plog/fcgi-vs-gunicorn-vs-uwsgi http://nichol.as/benchmark-of-python-web-servers

Can you restart the uwsgi instance(s) without restarting nginx? That's a pretty significant benefit of gunicorn...

Yes, uwsgi instances run separately from nginx. The uwsgi nginx module simply speaks a binary protocol for reverse proxying rather than HTTP.

Re: Python Deployment Anti-Patterns

#117

Post by nobleach »

smacktoward wrote: post #14

I don't get the negativity on using your distro's packages, at least from the staying-stable perspective. Any decent package manager should let you pin/hold critical packages on a particular version, so if "the next Ubuntu ships with a different SQLAlchemy by default" you just hold the SQLAlchemy package at the version you want and then ignore it until you're ready to make that move. 99% percent of the time when I he…

Oh the problem has NOTHING to do with "10 minutes ago".

Ubuntus's latest release 11.10 (yes I know 12.04 is a couple of days away) is Python 2.4. I don't remember what Ruby it is, but it's something like ~=1.8.7. Ruby is on 1.9.3 and the next version of Rails won't even support 1.8.7.

I'd be fine with a year or two old Python and Ruby....

Re: Python Deployment Anti-Patterns

#118

Post by dekkers »

djtriptych wrote: post #4

As a python dev who deploys a lot of software, I found this article to be wonderfully helpful and informative, and a good reflection of current best practices. Summary of the deployment tools mentioned: - Manage remote daemons with supervisord http://supervisord.org/ - Manage python packages with pip (and use `pip freeze`) http://pypi.python.org/pypi/pip http://www.pip-installer.org/en/latest/requirements.html - Mana…

I use gevent instead of twisted for event-based python. Gevent is a lot nicer to work with and doesn't make your code less readable in the way twisted does with its callbacks and errbacks. It's also a lot easier to do unit testing with gevent.

Re: Python Deployment Anti-Patterns

#119

Post by davvid »

nobleach wrote: post #117
smacktoward wrote: post #14

I don't get the negativity on using your distro's packages, at least from the staying-stable perspective. Any decent package manager should let you pin/hold critical packages on a particular version, so if "the next Ubuntu ships with a different SQLAlchemy by default" you just hold the SQLAlchemy package at the version you want and then ignore it until you're ready to make that move. 99% percent of the time when I he…

Oh the problem has NOTHING to do with "10 minutes ago". Ubuntus's latest release 11.10 (yes I know 12.04 is a couple of days away) is Python 2.4. I don't remember what Ruby it is, but it's something like ~=1.8.7. Ruby is on 1.9.3 and the next version of Rails won't even support 1.8.7. I'd be fine with a year or two old Python and Ruby....

Huh?

    $ uname -a
    Linux apollo 3.0.0-17-generic #30-Ubuntu SMP Thu Mar 8 20:45:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

    $ python -V
    Python 2.7.2+

Re: Python Deployment Anti-Patterns

#120

Post by pconf »

mdehaan wrote: post #41
djtriptych wrote: post #4

As a python dev who deploys a lot of software, I found this article to be wonderfully helpful and informative, and a good reflection of current best practices. Summary of the deployment tools mentioned: - Manage remote daemons with supervisord http://supervisord.org/ - Manage python packages with pip (and use `pip freeze`) http://pypi.python.org/pypi/pip http://www.pip-installer.org/en/latest/requirements.html - Mana…

Chef requires Ruby programming. Puppet doesn't, but the core is obviously Ruby, and you extend it using Ruby. You may possibly like my new project: http://ansible.github.com The core is Python but you can write modules in any language.

Thanks for that! We gave up on chef when one of their version updates failed to work with a prior version, both of which were the OS package defaults. Chef silently failed, no error message, nothing in the docs, nothing even in the source code. Had to do a fair bit of searching to find out why.

When open source projects like chef have nobody interested in even documenting much less testing backwards incompatibilities we move them to the bottom of our to-eval list.

This also illustrates a problem in article's blind enthusiasm for the latest revisions and libraries i.e., it dismisses the headaches this causes end-users, who often don't have staff or budget to fix whatever breaks during an upgrade. That said we are at least talking about python, which has had better release QA and backwards compatibility than perl, ruby or, gasp, php.

Post reply on HN