Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Deploy Python Code

nylas.com

Post reply on HN
Print view View original

51–60 of 129 posts

Re: How We Deploy Python Code

#51

Post by vacri »

x0x0 wrote: post #28
jedberg wrote: post #8

Earlier quoted context omitted.

There was a whole paragraph in the article about why Docker didn't work for them.

they had 3 reasons: one of which was just silly (kernel version -- are you living on that point release forever?) one of which was valid (necessity to maintain method for distributing docker images), but probably dumb: you only get so many innovation points per company, and innovating on a problem docker just solves means you are supporting your in-house solution ad infinitum and one of which definitely sounds painfu…

The 3.2 kernel they mention is the standard kernel of Ubuntu 12.04. I have some docker machines of the same vintage, and it's as simple as installing a backported kernel from the official repos.

This being said, I'm using docker for packaging/deployment of a nodejs app on those machines, and I hate it. I'm about to strip it out and go for .debs. Docker brings a lot of baggage with it, and requires major restructuring of some infrastructure parts. As they say in the article, the changes required to bring docker in just to do packaging are way too heavy. And Docker also sucks for rollbacks, to be honest - their tagging system is downright terrible.

My advice is not to use Docker in a production environment unless you can articulate the specific pain points it will solve for you.

Re: How We Deploy Python Code

#52

Post by zobzu »

Cieplak wrote: post #3

Highly recommend FPM for creating packages (deb, rpm, osx .pkg, tar) from gems, python modules, and pears. https://github.com/jordansissel/fpm

fpm is cool - very cool. that said it wont create proper spec files and friends, thus its harder to maintain the packages properly. also they usually ont get accepted upstream due to that.

that said, for python files and simple packages it works well enough!

Re: How We Deploy Python Code

#53

Post by webo »

stephenr wrote: post #47
webo wrote: post #46

> Building with dh-virtualenv simply creates a debian package that includes a virtualenv, along with any dependencies listed in the requirements.txt file. So how is this solving the first issue? If PyPI or the Git server is down, this is exactly like the git & pip option.

You need those things up to build the package not to install it

Ah I misunderstood the article. I just package my application source in a tar during deployment. I thought that's what most people do.

Re: How We Deploy Python Code

#54

Post by ytjohn »

BuckRogers wrote: post #42

Seems this method wouldn't work as well if you have external clients you deploy for. I'd use Docker instead of doing this, just to be in a better position for an internal or external client deployment.

If you took this a step further and set up a debian repo, then you could have your clients use that debian repo.

I'm looking to do something pretty similiar, but RPMs. I found rpmvenv that seems to work in the same fashion. https://pypi.python.org/pypi/rpmvenv/0.3.1

Re: How We Deploy Python Code

#55

Post by tuckermi »

TDL wrote: post #12
ah- wrote: post #4

conda works pretty well.

Agreed (although biased since I used to work at Continuum.) I am wondering what others think of conda?

My team has been rolling our own conda packages for (frequent) internal software releases to local servers and have been pretty happy overall pulling down code from a locally managed conda package repo.

With that said, Conda is not a perfect solution. One thing that can be frustrating is that a package can include compiled code (shared objects/dylibs) that may be incompatible with your system. Unfortunately, while you can indicate dependencies on other conda packages, python versions, etc there isn't currently a convenient way to indicate things like GLIBC dependencies.

Re: How We Deploy Python Code

#56

Post by doki_pen »

We do something similar at embedly, except instead of dh-virtualenv we have our own homegrown solution. I wish I new about dh-virtualenv before we created it.

Basically, what it comes down to a build script that builds a deb with the virtualenv of your project versioned properly(build number, git tag), along with any other files that need to be installed (think init scripts and some about file describing the build). It also should do things like create users for daemons. We also use it to enforce consistent package structure.

We use devpi to host our python libraries (as opposed to applications), reprepro to host our deb packages, standard python tools to build the virtualenv and fpm to package it all up into a deb.

All in all, the bash build script is 177 LoC and is driven by a standard build script we include in every applications repository defining variables, and optionally overriding build steps (if you've used portage...).

The most important thing is that you have a standard way to create python libraries and application to reduce friction on starting new projects and getting them into production quickly.

Re: How We Deploy Python Code

#57

Post by avilay »

webo wrote: post #50
avilay wrote: post #45

Here is the process I use for smallish services - 1. Create a python package using setup.py 2. Upload the resulting .tar.gz file to a central location 3. Download to prod nodes and run pip3 install .tar.gz Rolling back is pretty simple - pip3 uninstall the current version and re-install the old version. Any gotchas with this process?

You have to do this every time there's a change in the codebase which is not easy. How do you stick this into a CI without the git & pip issue talked about in the post?

I have to do this everytime I have to deploy, which is similar to having to create a deb package everytime Nylas has to deploy.

There are no git dependencies in the process I describe above.

The pip drawback that is discussed in the post is of PyPi going down. In the process described above there is no PyPi dependency. Storing the .tar.gz package in a central location is similar to Nylas storing their deb package on S3.

Re: How We Deploy Python Code

#58

Post by rfeather »

I've had decent results using a combination of bamboo, maven, conda, and pip. Granted, most of our ecosystem is Java. Tagging a python package along as a maven artifact probably isn't the most natural thing to do otherwise.

Re: How We Deploy Python Code

#59

Post by doki_pen »

emidln wrote: post #31
vamega wrote: post #30

Earlier quoted context omitted.

Can you point me to your recommended PyPI-in-a-box system?

I currently use this one: https://localshop.readthedocs.org/en/latest/installing.html It works. It's django based and you can setup s3-backed storage. It also has a docker-compose script.

We migrated off of localshop and onto devpi. Devpi is a much better product and much more actively maintained. localshop was nothing but headaches and constantly breaking.

Re: How We Deploy Python Code

#60

Post by doki_pen »

StavrosK wrote: post #41

Unfortunately, this method seems like it would only work for libraries, or things that can easily be packaged as libraries. It wouldn't work that well for a web application, for example, especially since the typical Django application usually involves multiple services, different settings per machine, etc.

I use a similar method to ship app.embed.ly (emberapp on top of django). You can package whatever you want in a deb. The virtualenv is just a part of it. Config files are managed with configuration management system (chef in our case). Our django settings.py file just tries to import from /etc/blahblah
Post reply on HN