Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Deploy Python Code

nylas.com

Post reply on HN
Print view View original

61–70 of 129 posts

Re: How We Deploy Python Code

#61

Post by dwb »

sandGorgon wrote: post #5

The fact that we had a weird combination of python and libraries took us towards Docker. And we have never looked back. For someone trying out building python deployment packages using deb, rpm, etc. I really recommend Docker.

We love Docker where I work, but running it in production is a big challenge — there's no "just" using it. We build packages of Rails apps and dependencies quite happily. Sure, you have to make sure all dependencies are packaged too, but that's still easier than a full-on Docker roll-out.

Indeed, we actually use Docker to build packages. Blog post coming soon, maybe.

Re: How We Deploy Python Code

#62

Post by acdha »

vacri wrote: post #44
StavrosK wrote: post #41

Unfortunately, this method seems like it would only work for libraries, or things that can easily be packaged as libraries. It wouldn't work that well for a web application, for example, especially since the typical Django application usually involves multiple services, different settings per machine, etc.

> different settings per machine /etc/default/mycoolapp.conf Debian packages have the concept of 'config' files. Files will be automatically overwritten when installing a new version of package FOO, unless they're marked as config files in the .deb manifest. This allows you to have a set of sane defaults, but not to lose customisations when upgrading.

Just wanted to +1 this. There are literally decades of convention built around patterns where you ship a standard config file which merges in system/user/instance-specific settings from a known location, command-line argument, environment, etc. The Debian world in particular has lead the community for decades with the use of debconf to store values such as a hostname or server role which can automatically be re-applied when otherwise unmodified files are modified upstream.

When I used this approach with a Django site years ago using RPM[1] we used the pattern vacri mentioned or the reverse one where you have an Apache virtualhost file which contains system-specific settings (hostname, SSL certs, log file name, etc.) and simply included the generic settings shipped in the RPM.

In either case the system-specific information can be set by hand (this was a .gov server…), managed with your favorite deployment / config tool, etc. and allows you to use the same signed, bit-for-bit identical package on testing, staging, and production with complete assurance that the only differences were intentional. This was really nice when you wanted to hand things off to a different group rather than having the dev team include the sysadmins.

1. http://chris.improbable.org/2009/10/16/deploying-django-site...

Re: How We Deploy Python Code

#63

Post by nZac »

We just commit our dependencies into our project repository in wheel format and install into a virtual env on prod from that directory eliminating PyPi. Though I don't know many other that do this. Do you?

Bitbucket and GitHub are reliable enough for how often we deploy that we aren't all that worried about downtime from those services. We could also pull from a dev's machine should the situation be that dire.

We have looked into Docker but that tool has a lot more growing before "I" would feel comfortable putting it into production. I would rather ship a packaged VM than Docker at this point, there are to many gotchas that we don't have time to figure out.

Re: How We Deploy Python Code

#64

Post by emidln »

vamega wrote: post #39
emidln wrote: post #31

Earlier quoted context omitted.

I currently use this one: https://localshop.readthedocs.org/en/latest/installing.html It works. It's django based and you can setup s3-backed storage. It also has a docker-compose script.

Do I understand this correctly? It only mirrors the packages that are requested from it? So I won't need to download 100GB+ of packages that I am not interested in?

Correct. My team has two main use cases: private packages and guaranteed access to packages we've built with. It's extremely frustrating to come into a codebase after several months or years to find it using a library that no longer seems to exist on the public internet.

Re: How We Deploy Python Code

#65

Post by emidln »

doki_pen wrote: post #59
emidln wrote: post #31

Earlier quoted context omitted.

I currently use this one: https://localshop.readthedocs.org/en/latest/installing.html It works. It's django based and you can setup s3-backed storage. It also has a docker-compose script.

We migrated off of localshop and onto devpi. Devpi is a much better product and much more actively maintained. localshop was nothing but headaches and constantly breaking.

Maybe you use more esoteric features. The only thing I've done in the last 18 months is patch a bug that prevented uploads from Python 2.7.4-2.7.10. We just run it under circus with chaussette and front it with an elb.

Re: How We Deploy Python Code

#66

Post by so0k »

nZac wrote: post #63

We just commit our dependencies into our project repository in wheel format and install into a virtual env on prod from that directory eliminating PyPi. Though I don't know many other that do this. Do you? Bitbucket and GitHub are reliable enough for how often we deploy that we aren't all that worried about downtime from those services. We could also pull from a dev's machine should the situation be that dire. We hav…

we download to a folder on the docker build server and build docker containers from this cache.

see here: http://stackoverflow.com/a/29936384/138469

Re: How We Deploy Python Code

#67

Post by tspike »

StavrosK wrote: post #41

Unfortunately, this method seems like it would only work for libraries, or things that can easily be packaged as libraries. It wouldn't work that well for a web application, for example, especially since the typical Django application usually involves multiple services, different settings per machine, etc.

Nah, the approach is sound. We did this for a Flask app that used celery, redis, etc and we were happy with it. For the software, use a .deb, for the configuration, use a configuration management tool like Ansible.

Re: How We Deploy Python Code

#69

Post by jacques_chester »

dwb wrote: post #61
sandGorgon wrote: post #5

The fact that we had a weird combination of python and libraries took us towards Docker. And we have never looked back. For someone trying out building python deployment packages using deb, rpm, etc. I really recommend Docker.

We love Docker where I work, but running it in production is a big challenge — there's no "just" using it. We build packages of Rails apps and dependencies quite happily. Sure, you have to make sure all dependencies are packaged too, but that's still easier than a full-on Docker roll-out. Indeed, we actually use Docker to build packages. Blog post coming soon, maybe.

In a few months Cloud Foundry will natively support launching, placing, managing, wiring, routing, logging and servicing Dockerised application images.

In the meantime you can get a taste with Lattice[0].

[0] http://lattice.cf/

Re: How We Deploy Python Code

#70

Post by ronjouch »

vamega wrote: post #30
emidln wrote: post #21

Earlier quoted context omitted.

Pair this with virtualenvs in separate directories (so that "rollback" is just a ssh mv and a reload for whatever supervisor process) and you get to skip the mess of building system packages. Also, are there seriously places that don't run their own PyPI mirrors? Places that have people who understand how to integrate platform-specific packages but can't be bothered to deploy one of the several PyPI-in-a-box systems…

Can you point me to your recommended PyPI-in-a-box system?

As a first step, what about trying pip's `-f` option, combined with dumping your wheels in a dumb folder served by apache/nginx:

    pip --help
    [...]
    Install Options:
      [...]
      -f, --find-links       If a url or path to an html file, then parse for links to archives. If a local path or file:// url that's a directory,then look for archives in
                                  the directory listing.
EDIT: in the context of deploying some app at work, what's the interest of a full-blown hosted cheeseshop? Users of these solutions, what value does it add to a simple `pip install -f INTERNAL_PKG_URL pkg_a==1.2.3 pkg_b==0.1.2` ? Which features do you frequently use?
Post reply on HN