How We Deploy Python Code • Hacker News Board

Re: How We Deploy Python Code

#81

Post by morgante » Fri, Jul 10, 2015, 3:29 AM UTC

viraptor wrote: post #79

morgante wrote: post #75

Their reason for dismissing Docker are rather shallow, considering that it's pretty much the perfect solution to this problem. Their first reason (not wanting to upgrade a kernel) is terrible considering that they'll eventually be upgrading it anyways. Their second is slightly better, but it's really not that hard. There are plenty of hosted services for storing Docker images, not to mention that "there's a Dockerfil…

> (not wanting to upgrade a kernel) is terrible considering that they'll eventually be upgrading it anyways. If they're running ubuntu 12.04 LTS they can keep the 3.2 kernel until late 2017. That's 2 more years. And they wrote "did not", so it was likely the situation months ago, not yesterday. > (not wanting to learn and convert to a new infrastructure paradigm) is the most legitimate, but ultimately misguided It de…

> If they handle everything using Ansible (and from the list it looks like they do), then it's months of work to migrate to something else.

It's not. It would be months of work if they wanted to convert all their Ansible code to Docker, but that's by no means required.

Docker and Ansible can easily coexist peacefully.

Re: How We Deploy Python Code

#82

Post by viraptor » Fri, Jul 10, 2015, 3:30 AM UTC

> curl “https://artifacts.nylas.net/sync-engine-3k48dls.deb” -o $temp ; dpkg -i $temp

It's really not hard to deploy a package repository. Either a "proper" one with a tool like `reprepro`, or a stripped one which is basically just .deb files in one directory. There's really no need for curl+dpkg. And a proper repository gives you dependency handling for free.

Re: How We Deploy Python Code

#83

Post by daurnimator » Fri, Jul 10, 2015, 3:37 AM UTC

Cieplak wrote: post #3

Highly recommend FPM for creating packages (deb, rpm, osx .pkg, tar) from gems, python modules, and pears. https://github.com/jordansissel/fpm

Rather than such a blunt tool like fpm, if you're deploying only python, look to something like py2deb: https://github.com/paylogic/py2deb

Re: How We Deploy Python Code

#84

Post by pyre » Fri, Jul 10, 2015, 3:45 AM UTC

x0x0 wrote: post #26

craigmccaskill wrote: post #11

Earlier quoted context omitted.

They specifically called that out in the article with an entire section called "just use docker".

if you have to choose between a kernel and docker, just choose docker. Python can't get their shit together deployment-wise, and docker is the one true route (tm) to python deployment happiness. forget virtualenv; forget package dependencies on conflicting versions of libxml; forget coworkers that have 3 different conflicting versions of requests scattered through various services, and goddamnit I just want to run a…

Ha. Wait until you need to run a build of shared Perl codebase against unit tests in all of the dependent codebases... but some of those codebases compile and run C (or C++) programs... and some of those codebases depend on conflicting versions of GCC!

"If we hit the bullseye, the rest of the dominos will fall like a house of cards... checkmate!" -- Zap Brannigan

> forget coworkers that scribble droppings all over the filesystem, and assume certain services will never coexist

I think this tends to be less of a problem than the desire to have a build artifact that can be reliably deployed to multiple servers, rather than having the "build" process and "deploy" process hopelessly intertwined with each other.

Re: How We Deploy Python Code

#85

Post by viraptor » Fri, Jul 10, 2015, 3:45 AM UTC

morgante wrote: post #81

viraptor wrote: post #79

Earlier quoted context omitted.

> (not wanting to upgrade a kernel) is terrible considering that they'll eventually be upgrading it anyways. If they're running ubuntu 12.04 LTS they can keep the 3.2 kernel until late 2017. That's 2 more years. And they wrote "did not", so it was likely the situation months ago, not yesterday. > (not wanting to learn and convert to a new infrastructure paradigm) is the most legitimate, but ultimately misguided It de…

> If they handle everything using Ansible (and from the list it looks like they do), then it's months of work to migrate to something else. It's not. It would be months of work if they wanted to convert all their Ansible code to Docker, but that's by no means required. Docker and Ansible can easily coexist peacefully.

They can. But depending on how you used Ansible before, it may mean a heavy rewrite of your deployment strategy. I'm not saying it will always take that long. But depending on your app, the requirements may be very complex and not fit into the docker idea.

(it always means some extra work for security updates though - now you're updating both the host and images)

Re: How We Deploy Python Code

#86

Post by viraptor » Fri, Jul 10, 2015, 3:53 AM UTC

mattbillenstein wrote: post #43

tschellenbach wrote: post #29

Yes, someone should build the one way to ship your app. No reason for everybody to be inventing this stuff over and over again. Deploys are harder if you have a large codebase to ship. rSync works really well in those cases. It requires a bit of extra infrastructure, but is super fast.

+1 rsync is pretty darn good at any scale -- I'm not sure why the simplest solution possible doesn't beat out docker as a suggestion in this thread. I've been bundling libs and software into a single virtual environment like package that I distribute with rsync for a long time - it solves loads of problems, is easy to bootstrap a new system with, and incremental updates are super fast. Combine that with rsync distrib…

Rsync is good for simple things. But it will fail with more complicated apps:

- how do you know which version you're running right now?

- how do you deploy to two environments where different deps are needed?

- how do you tell when your included dependencies need security patches?

Re: How We Deploy Python Code

#87

Post by sciurus » Fri, Jul 10, 2015, 4:09 AM UTC

viraptor wrote: post #82

> curl “ https://artifacts.nylas.net/sync-engine-3k48dls.deb” -o $temp ; dpkg -i $temp It's really not hard to deploy a package repository. Either a "proper" one with a tool like `reprepro`, or a stripped one which is basically just .deb files in one directory. There's really no need for curl+dpkg. And a proper repository gives you dependency handling for free.

Yes, that part really surprised me. Barebones repos are useful enough, and there's also some pretty fancy tools out there like http://www.aptly.info/

Re: How We Deploy Python Code

#88

Post by pekk » Fri, Jul 10, 2015, 4:35 AM UTC

TDL wrote: post #12

ah- wrote: post #4

conda works pretty well.

Agreed (although biased since I used to work at Continuum.) I am wondering what others think of conda?

What was wrong with Python packaging that conda needed to replace it entirely (including virtualenv)? If you say the installation story for the scipy stack, that is less a matter of Python packaging and more a matter of the scipy stack.

Re: How We Deploy Python Code

#89

Post by stephenr » Fri, Jul 10, 2015, 4:49 AM UTC

tschellenbach wrote: post #29

Yes, someone should build the one way to ship your app. No reason for everybody to be inventing this stuff over and over again. Deploys are harder if you have a large codebase to ship. rSync works really well in those cases. It requires a bit of extra infrastructure, but is super fast.

They have. Every mainstream OS/Distro has a packaging system and an installer for those packages.

For server-side apps like this, that usually means a Deb or an RPM. These systems handle upgrades, rollbacks, dependencies, etc.

Just because some people decide that writing an RPM specfile or running dh_make is too hard to work out, doesn't mean that the solution doesn't exist.

Re: How We Deploy Python Code

#90

Post by stephenr » Fri, Jul 10, 2015, 4:52 AM UTC

velocitypsycho wrote: post #73

For installing using .deb files, how are db migrations handled. Our deployment system handles running django migrations by deploying to a new folder/virtualenv, running the migrations, then switching over symlinks. I vaguely remember .deb files having install scripts, is that what one would use?

Depends on your infra. If it's a single server with the app + Db (or a single app server + single DB server) you could have a postinst script that calls your app/framework's migration system.

If your migration system is smart enough (or you can easily check the migration status from a shell script) you could also do this in a multi-app-server environment too.