Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Deploy Python Code

nylas.com

Post reply on HN
Print view View original

101–110 of 129 posts

Re: How We Deploy Python Code

#101

Post by thraxil »

svieira wrote: post #72

Back when I was doing Python deployments (~2009-2013) I was: * Downloading any new dependencies to a cached folder on the server (this was before wheels had really taken off) * Running pip install -r requirements.txt from that cached folder into a new virtual environment for that deployment (`/opt/company/app-name/YYYY-MM-DD-HH-MM-SS`) * Switching a symlink (`/some/path/app-name`) to point at the latest virtual env.…

This is basically how we still do it (with nginx + gunicorn rather than apache). Wheels (and recent setuptools/pip versions that build and cache them automatically) just made the build step significantly faster (lxml, argh).

We're moving to the Docker approach, which is really nice, but it does change the shape of the whole deploy pipeline, so it's going to take some time.

Re: How We Deploy Python Code

#102

Post by objectified »

I recently created vdist (https://vdist.readthedocs.org/en/latest/ - https://github.com/objectified/vdist) for doing similar things - the exception being is that it uses Docker to actually build the OS package on. vdist uses FPM under the hood, and (currently) lets you build both deb and rpm packages. It also packs up a complete virtualenv, and installs the build time OS dependencies on the Docker machine where it builds on when needed. The runtime dependencies are made into dependencies of the resulting package.

Re: How We Deploy Python Code

#103

Post by crdoconnor »

svieira wrote: post #72

Back when I was doing Python deployments (~2009-2013) I was: * Downloading any new dependencies to a cached folder on the server (this was before wheels had really taken off) * Running pip install -r requirements.txt from that cached folder into a new virtual environment for that deployment (`/opt/company/app-name/YYYY-MM-DD-HH-MM-SS`) * Switching a symlink (`/some/path/app-name`) to point at the latest virtual env.…

This is exactly what I do.

>Use a local PyPi rather than a per-server cache

I stil prefer a per-server cache. A local pypi is another piece of infrastructure you need to keep alive. You don't have to worry about the uptime of an rsync playbook.

Re: How We Deploy Python Code

#105

Post by xorcist »

vacri wrote: post #51
x0x0 wrote: post #28

Earlier quoted context omitted.

they had 3 reasons: one of which was just silly (kernel version -- are you living on that point release forever?) one of which was valid (necessity to maintain method for distributing docker images), but probably dumb: you only get so many innovation points per company, and innovating on a problem docker just solves means you are supporting your in-house solution ad infinitum and one of which definitely sounds painfu…

The 3.2 kernel they mention is the standard kernel of Ubuntu 12.04. I have some docker machines of the same vintage, and it's as simple as installing a backported kernel from the official repos. This being said, I'm using docker for packaging/deployment of a nodejs app on those machines, and I hate it. I'm about to strip it out and go for .debs. Docker brings a lot of baggage with it, and requires major restructuring…

Agreed on the rollback part.

It is easy to pick the silliness of the kernel reason, but Docker is moving fast right now. They are still getting the basic building blocks in place, and the Docker in two years will look nothing like today's.

We use Docker quite a bit today, but it's immature and it shows. With Composer I feel the basic functionality is finally in place it needs time to mature.

So I think it's quite wise to wait. You don't need to chase every new technology. If you have a product to ship, focus on that instead and use whatever tools are proven to work.

Re: How We Deploy Python Code

#106

Post by mattbillenstein »

viraptor wrote: post #86
mattbillenstein wrote: post #43

Earlier quoted context omitted.

+1 rsync is pretty darn good at any scale -- I'm not sure why the simplest solution possible doesn't beat out docker as a suggestion in this thread. I've been bundling libs and software into a single virtual environment like package that I distribute with rsync for a long time - it solves loads of problems, is easy to bootstrap a new system with, and incremental updates are super fast. Combine that with rsync distrib…

Rsync is good for simple things. But it will fail with more complicated apps: - how do you know which version you're running right now? - how do you deploy to two environments where different deps are needed? - how do you tell when your included dependencies need security patches?

rsync isn't the complete system - you're going to need git (or another vcs) and some other tools of course.

#1 is git (dump and log the git head on a deploy) #2 don't do that - keep a single consistent environment #3 use the system openssl - monitor other software components for security updates -- you need to do this anyway in any of these systems.

Re: How We Deploy Python Code

#107

Post by viraptor »

mixmastamyk wrote: post #97
viraptor wrote: post #79

Earlier quoted context omitted.

> (not wanting to upgrade a kernel) is terrible considering that they'll eventually be upgrading it anyways. If they're running ubuntu 12.04 LTS they can keep the 3.2 kernel until late 2017. That's 2 more years. And they wrote "did not", so it was likely the situation months ago, not yesterday. > (not wanting to learn and convert to a new infrastructure paradigm) is the most legitimate, but ultimately misguided It de…

sudo apt-get install linux-generic-lts-quantal Easy to upgrade.

If you can, it's easy. I wish everyone to work in environments where it's that simple ;)

Re: How We Deploy Python Code

#108

Post by viraptor »

mixmastamyk wrote: post #95
viraptor wrote: post #82

> curl “ https://artifacts.nylas.net/sync-engine-3k48dls.deb” -o $temp ; dpkg -i $temp It's really not hard to deploy a package repository. Either a "proper" one with a tool like `reprepro`, or a stripped one which is basically just .deb files in one directory. There's really no need for curl+dpkg. And a proper repository gives you dependency handling for free.

Could you elaborate on the simple folder? For example I find the --instdir option to dpkg but it still would have to be downloaded from the other host, unless of course the folder was mounted somehow.

Search for debian's "trivial archive". It replaces release/component elements with explicit path. It's deprecated now, but I believe still works.

Re: How We Deploy Python Code

#109

Post by viraptor »

mattbillenstein wrote: post #106
viraptor wrote: post #86

Earlier quoted context omitted.

Rsync is good for simple things. But it will fail with more complicated apps: - how do you know which version you're running right now? - how do you deploy to two environments where different deps are needed? - how do you tell when your included dependencies need security patches?

rsync isn't the complete system - you're going to need git (or another vcs) and some other tools of course. #1 is git (dump and log the git head on a deploy) #2 don't do that - keep a single consistent environment #3 use the system openssl - monitor other software components for security updates -- you need to do this anyway in any of these systems.

> #2 don't do that

I wish everyone to have easy deployments where environments, OS versions and everything else are always consistent. :)

> #3 monitor other software components for security updates -- you need to do this anyway in any of these systems.

Sure. But having multiple virtualenvs means you need to monitor all of them on all of deployed hosts. Having everything packaged separately means you can do audits much easier and without location-specific checks.

Re: How We Deploy Python Code

#110

Post by emidln »

viraptor wrote: post #74
emidln wrote: post #21

Earlier quoted context omitted.

Pair this with virtualenvs in separate directories (so that "rollback" is just a ssh mv and a reload for whatever supervisor process) and you get to skip the mess of building system packages. Also, are there seriously places that don't run their own PyPI mirrors? Places that have people who understand how to integrate platform-specific packages but can't be bothered to deploy one of the several PyPI-in-a-box systems…

> you get to skip the mess of building system packages. Only in cases where you don't have wheels depending on external libraries. If you do, you should still package with the right dependency constraints. Otherwise you can install a wheel which does not work (because of missing .so)

This simply isn't true. You can package .so or .dll (or anything else) with your wheel. An example of doing it with a dll or so is here: http://stackoverflow.com/questions/24071491/how-can-i-make-a...
Post reply on HN