Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deploying at GitHub

github.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Deploying at GitHub

#31

Post by shabble »

xxpor wrote: post #10
wamatt wrote: post #6

Encouraging to know this model scales to 100 employees at least. Purely out of intellectual interest, I wonder if a company the size of Google or Facebook could also ship in this way, or if the whole release manager/team is essential.

Amazon generally doesn't have release teams (except for the retail website). I can't find it right now, but I know someone gave a talk on Apollo, which is how Amazon does deployments.

I think I know the thing you mean, but I can't find it either.

I did find 'flippers' from Flickr[1] which sounds like a very similar "keep it all in master and turn stuff on and off" methodology.

[1] http://code.flickr.com/blog/2009/12/02/flipping-out/

Re: Deploying at GitHub

#32

Post by technoweenie »

amccloud wrote: post #20

If GitHub uses GitHub to deploy GitHub, what happens when GitHub goes down?

Heaven (our deployment tool) does deploy GitHub.com directly from the file servers. But, most of our infrastructure directly relies on GitHub too (such as the Merge API from the blog post, service hooks, and a bunch of OAuth minapps).

Re: Deploying at GitHub

#33

Post by technoweenie »

vrish88 wrote: post #29
psadauskas wrote: post #21

Earlier quoted context omitted.

We have a staging environment, but its really only used for really big changes that might need to be experimented with before being deployed. We can also deploy a branch to a single front end to observe how it behaves with a subset of the traffic, and roll it back quickly if needed. Also, most large user-facing features are released as "staff-only" first, so we as GitHub users are able to play around with it for a fe…

How do you guys release features as "staff-only"? Do you have some internal tool that manages that?

We do use Rollout (https://github.com/jamesgolick/rollout) once in awhile. Most of the time, we like having the history of flipped feature flags in the Git code though.

Re: Deploying at GitHub

#34

Post by technoweenie »

46Bit wrote: post #26
tzaman wrote: post #14

WOW . 175 deploys in one day?

I'd love to hear the backstory behind this. Company hackday, or a day spent shipping a major set of new features?

We don't do "company hack days". If you feel like hacking on something, hack on it.

We do have days where multiple people will be waiting in line waiting for their chance to deploy their tweak.

That particular day consisted of staff deploys on multiple in-progress branches, some performance tuning, bug fixes, etc. Nothing crazy.

I'm also quite sure the number counts deploys across all of our applications. For instance, deploying a change to github-services counts as two, since I have to deploy changes to GitHub.com also.

Re: Deploying at GitHub

#35

Post by technoweenie »

smg wrote: post #13

How do you deal with the github enterprise version of your software? Does it have a separate QA cycle? How often do you ship new releases of that? I am hoping that Github could shed more light on the how they ship an enterprise version along with the SAASy web version that we all know and love.

I'm prodding the Enterprise team to blog about this :)

Re: Deploying at GitHub

#36

Post by jedberg »

jrockway wrote: post #12
wamatt wrote: post #6

Encouraging to know this model scales to 100 employees at least. Purely out of intellectual interest, I wonder if a company the size of Google or Facebook could also ship in this way, or if the whole release manager/team is essential.

Sarbanes-Oxley puts a big damper on production deployments at big companies. I don't fully understand it so I won't try to explain it. (I will complain though: the law says developers shouldn't have control over production systems. If that's a requirement, who's going to write the software?)

It hasn't slowed us down at Netflix. :) In fact what it has done is caused us to be really good about separating what needs auditing from what doesn't, so that only a very minimal set of services has to have separations and release processes that are in line with SOX controls.

Re: Deploying at GitHub

#37

Post by jrockway »

jedberg wrote: post #36
jrockway wrote: post #12

Earlier quoted context omitted.

Sarbanes-Oxley puts a big damper on production deployments at big companies. I don't fully understand it so I won't try to explain it. (I will complain though: the law says developers shouldn't have control over production systems. If that's a requirement, who's going to write the software?)

It hasn't slowed us down at Netflix. :) In fact what it has done is caused us to be really good about separating what needs auditing from what doesn't, so that only a very minimal set of services has to have separations and release processes that are in line with SOX controls.

That's good to hear.

Re: Deploying at GitHub

#38

Post by 46Bit »

technoweenie wrote: post #34
46Bit wrote: post #26

Earlier quoted context omitted.

I'd love to hear the backstory behind this. Company hackday, or a day spent shipping a major set of new features?

We don't do "company hack days". If you feel like hacking on something, hack on it. We do have days where multiple people will be waiting in line waiting for their chance to deploy their tweak. That particular day consisted of staff deploys on multiple in-progress branches, some performance tuning, bug fixes, etc. Nothing crazy. I'm also quite sure the number counts deploys across all of our applications. For instanc…

Thanks for this, enjoy hearing about Github as a company.

> I'm also quite sure the number counts deploys across all of our applications. For instance, deploying a change to github-services counts as two, since I have to deploy changes to GitHub.com also. That might explain a lot. Still a lot of deploys, but a more sane count :-)

Re: Deploying at GitHub

#39

Post by cam- »

tomjakubowski wrote: post #23
jrockway wrote: post #12

Earlier quoted context omitted.

Sarbanes-Oxley puts a big damper on production deployments at big companies. I don't fully understand it so I won't try to explain it. (I will complain though: the law says developers shouldn't have control over production systems. If that's a requirement, who's going to write the software?)

Wait, what? Could someone elaborate on S-O restricting developer control over production systems?

Everyone just 'knows' what is in Sarbanes Oxley but when you ask them to point it out to you in the legislation they cannot find what they were so certain about 2 minutes prior. We have compliance people and auditors are always coming in, but when someone claims something is required for Sox compliance, challenge them on it as 99% of the time it is a convention because someone told them, or they did it like that somewhere else once, rather then what is required by law. At the least it will make them justify the compliance/overhead they are causing you to do as an engineer.

Here is the legislation if you want to read through it or use it to challenge someone's assumptions about the Sarbanes-Oxley; http://www.sec.gov/about/laws/soa2002.pdf

Re: Deploying at GitHub

#40

Post by xxpor »

shabble wrote: post #31
xxpor wrote: post #10

Earlier quoted context omitted.

Amazon generally doesn't have release teams (except for the retail website). I can't find it right now, but I know someone gave a talk on Apollo, which is how Amazon does deployments.

I think I know the thing you mean, but I can't find it either. I did find 'flippers' from Flickr[1] which sounds like a very similar "keep it all in master and turn stuff on and off" methodology. [1] http://code.flickr.com/blog/2009/12/02/flipping-out/

Here we go:

http://news.ycombinator.com/item?id=2971521

Post reply on HN