Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How We Deploy Python Code

nylas.com

Post reply on HN
Print view View original

31–40 of 129 posts

Re: How We Deploy Python Code

#31

Post by emidln »

vamega wrote: post #30
emidln wrote: post #21

Earlier quoted context omitted.

Pair this with virtualenvs in separate directories (so that "rollback" is just a ssh mv and a reload for whatever supervisor process) and you get to skip the mess of building system packages. Also, are there seriously places that don't run their own PyPI mirrors? Places that have people who understand how to integrate platform-specific packages but can't be bothered to deploy one of the several PyPI-in-a-box systems…

Can you point me to your recommended PyPI-in-a-box system?

I currently use this one: https://localshop.readthedocs.org/en/latest/installing.html

It works. It's django based and you can setup s3-backed storage. It also has a docker-compose script.

Re: How We Deploy Python Code

#32

Post by stefantalpalaru »

> The state of the art seems to be ”run git pull and pray”

No, the state of the art where I'm handling deployment is "run 'git push' to a test repo where a post-update hook runs a series of tests and if those tests pass it pushes to the production repo where a similar hook does any required additional operation".

Re: How We Deploy Python Code

#33

Post by themartorana »

stefantalpalaru wrote: post #32

> The state of the art seems to be ”run git pull and pray” No, the state of the art where I'm handling deployment is "run 'git push' to a test repo where a post-update hook runs a series of tests and if those tests pass it pushes to the production repo where a similar hook does any required additional operation".

Git deployments work great if you're packing an image (AMI, Docker) using, say, Packer. But we only deploy "immutable" images, not code on to existing servers.

Re: How We Deploy Python Code

#34

Post by amatix »

jlees wrote: post #10
Cieplak wrote: post #3

Highly recommend FPM for creating packages (deb, rpm, osx .pkg, tar) from gems, python modules, and pears. https://github.com/jordansissel/fpm

That seems like a neat tool. I wonder if you could combine it with the sandboxing that dh-virtualenv provides to get the best of both worlds?

we use fpm & something like dh-virtualenv along exactly those lines. Helps us manage a complex mix of native/system-level dependencies (non-python) as well as python packages.

We also incorporate a set of meta packages which means we can have multiple codebase versions installed and switch the "active" one by installing the right version of the meta-package. There's also meta-packages for each service running off the same codebase, which deals with starting/stopping/etc.

Re: How We Deploy Python Code

#35

Post by jumpkick »

hobarrera wrote: post #23

> The state of the art seems to be ”run git pull and pray” Looks like these guys never heard of things like CI.

You had to read further.

This is the core of how we deploy code at Nylas. Our continuous integration server (Jenkins) runs dh-virtualenv to build the package, and uses Python’s wheel cache to avoid re-building dependencies.

Re: How We Deploy Python Code

#36

Post by kbrownlees »

vamega wrote: post #30
emidln wrote: post #21

Earlier quoted context omitted.

Pair this with virtualenvs in separate directories (so that "rollback" is just a ssh mv and a reload for whatever supervisor process) and you get to skip the mess of building system packages. Also, are there seriously places that don't run their own PyPI mirrors? Places that have people who understand how to integrate platform-specific packages but can't be bothered to deploy one of the several PyPI-in-a-box systems…

Can you point me to your recommended PyPI-in-a-box system?

I have had good success with http://doc.devpi.net/latest/

Re: How We Deploy Python Code

#38

Post by vamega »

kbrownlees wrote: post #36
vamega wrote: post #30

Earlier quoted context omitted.

Can you point me to your recommended PyPI-in-a-box system?

I have had good success with http://doc.devpi.net/latest/

That looks good. Might work exceptionally well for my needs. I want to have a PyPI mirror in an environment that doesn't have public internet access.

Running devpi in another environment and syncing the resulting repository should allow me to achieve what I want.

Re: How We Deploy Python Code

#39

Post by vamega »

emidln wrote: post #31
vamega wrote: post #30

Earlier quoted context omitted.

Can you point me to your recommended PyPI-in-a-box system?

I currently use this one: https://localshop.readthedocs.org/en/latest/installing.html It works. It's django based and you can setup s3-backed storage. It also has a docker-compose script.

Do I understand this correctly? It only mirrors the packages that are requested from it? So I won't need to download 100GB+ of packages that I am not interested in?

Re: How We Deploy Python Code

#40

Post by vacri »

rhelmer wrote: post #17
jlees wrote: post #10

Earlier quoted context omitted.

That seems like a neat tool. I wonder if you could combine it with the sandboxing that dh-virtualenv provides to get the best of both worlds?

I've used fpm to make rpm and deb packages that simply include a virtualenv, it works ok. One of the significant tradeoffs to this approach is you lose the carefully-crafted tree-of-dependencies that the distros favor, so it makes the package pretty much automatically unacceptable to package maintainers. However, being able to have install instructions that amount to "yum/apt-get install " is pretty great. I am hopin…

FPM is written as an in-house solution only. It's not intended to be used for making packages for official distro repositories for third-party users to pick up, and they suggest you use the distro-specified methods for those.
Post reply on HN