Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Evolution of Code Deploys at Reddit

redditblog.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: The Evolution of Code Deploys at Reddit

#21

Post by spladug »

siliconc0w wrote: post #19

It's nice to have a lot of options when managing deployments. Feature flags, canary deploys, A/B or green/blue, etc. Canaries are really nice to catch the majority of deploy issues (i.e 'it doesn't start up' errors, obvious exception spikes or performance regressions). Feature flags let you hand off control to individual product teams and encourage them to create 'bite sized' changes which can be flagged. Blue/green…

Feature flags are super useful. We didn't really have room to cover our use of them in this post, but if you're curious you can see the system we currently use for it here: https://github.com/reddit/reddit/blob/master/r2/r2/config/fe...

Re: The Evolution of Code Deploys at Reddit

#22

Post by pacaro »

spladug wrote: post #3
eriknstr wrote: post #2

I think it's strange how little attention this got here on HN. Submitted by one of the Reddit admins even. I'd ask you to do an AMA but I have no questions and besides you lot are pretty good at answering questions whenever they come up anyway so.

Thanks! Do feel free to ask if you think of something :)

How do you handle one way gates? Clearly at each deploy there are two different versions running concurrently, and as you make changes you do so knowing that, but as the system evolves their are points in (code) time that you can't go back to. Is this not a concern because you would never roll back that far?

Re: The Evolution of Code Deploys at Reddit

#23

Post by oblio »

spladug wrote: post #16
mschuster91 wrote: post #15

A question out of curiosity: why did you choose to write your deployment tools from scratch, instead of going with something like Jenkins? And, how do/did you provision new servers? By hand, or did you use something like Chef/Puppet?

> why did you choose to write your deployment tools from scratch, instead of going with something like Jenkins? Each step along the way was basically just a small modification on the system before. AFAIK Jenkins doesn't come with the ability to safely deploy code to hundreds of servers out of the box, so building out the systems to make that the case would've been more work than just adding to what existed and for un…

Jenkins would have bought you deployment queueing, which I see you developed.

On the other hand, since you weren't already using it for builds/running tests, it would have added some overhead.

Re: The Evolution of Code Deploys at Reddit

#28

Post by nameless912 »

juanbrein wrote: post #26

Lot of the features you built over the years were built in tools like capistrano or fabric. Any particular reason on why not use them in the first place?

Probably simple-is-usually-better.

I've been doing an eval for a new environmental auditing tool at work, and I've found that most of the pre-built solutions out there (e.g. Ansible Tower, Chef Server, some tools that we have written internally) will mostly meet our needs with some coercion, but I decided we should write our own anyway because it gives us the flexibility to only use and maintain the features we're actually going to use.

It's very possible (likely, even) that the Reddit guys looked at fabric or capistrano, and decided either:

1) the tool didn't map to their model of deployments, or 2) the tool did too much and would require more maintenance than a dead-simple solution they wrote themselves.

It's all a matter of perspective.

Re: The Evolution of Code Deploys at Reddit

#29

Post by spladug »

nameless912 wrote: post #28
juanbrein wrote: post #26

Lot of the features you built over the years were built in tools like capistrano or fabric. Any particular reason on why not use them in the first place?

Probably simple-is-usually-better. I've been doing an eval for a new environmental auditing tool at work, and I've found that most of the pre-built solutions out there (e.g. Ansible Tower, Chef Server, some tools that we have written internally) will mostly meet our needs with some coercion, but I decided we should write our own anyway because it gives us the flexibility to only use and maintain the features we're ac…

Spot on.

Re: The Evolution of Code Deploys at Reddit

#30

Post by spladug »

pacaro wrote: post #22
spladug wrote: post #3

Earlier quoted context omitted.

Thanks! Do feel free to ask if you think of something :)

How do you handle one way gates? Clearly at each deploy there are two different versions running concurrently, and as you make changes you do so knowing that, but as the system evolves their are points in (code) time that you can't go back to. Is this not a concern because you would never roll back that far?

Yeah, rollbacks are more of adding a revert to the top of the pile so we just make sure we roll back things that can be rolled back. This is important to think about when planning deploys.
Post reply on HN