Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Evolution of Code Deploys at Reddit

redditblog.com

Post reply on HN
Print view View original

11–20 of 32 posts

Re: The Evolution of Code Deploys at Reddit

#11

Post by spladug »

eriknstr wrote: post #4
spladug wrote: post #3

Earlier quoted context omitted.

Thanks! Do feel free to ask if you think of something :)

Actually I came to think of something but it's not an AMA kind of question but a Reddit question though. I just opted in to the profile beta literally less than an hour ago and I like it but I have a bug report [1] and a feature request / feedback [2] that I'd like to be seen by the specific people that work on the profile portion of Reddit. Who do I contact about that? Or was submitting it to /r/beta sufficient and…

Just checked: r/beta is the right place for that. Thanks for the feedback!

Re: The Evolution of Code Deploys at Reddit

#12

Post by wyc »

Kudos to the Reddit engineering team. I've been really enjoying their quality posts as of late, and I wish more companies of that scale were as transparent with their engineering problems and solutions. Thank you.

Re: The Evolution of Code Deploys at Reddit

#13

Post by spladug »

jameskegel wrote: post #10
spladug wrote: post #9

Earlier quoted context omitted.

> Do you find yourself browsing Reddit in your spare time or do you avoid it? I personally still spend plenty of time reading stuff on the site, both educational (r/askhistorians, various programming subreddits) and entertainment (r/popular). I'll definitely tend to avoid the more meta subreddits when not in a work mood though. > Do you ever find yourself spending whole days at work without being on the site because…

I'd like to pose a follow up question; as your role at the company has progressed, do you feel your day to day (collective you, the [a]'s) is less development oriented and more social management of the different issues that arise when a community gets that large?

Actually quite the opposite. One of the benefits of being a larger company is separation of concerns. Our community team is a bunch of fantastic people that are focused on that side of things and engineering spend most of its time writing code. Obviously there will be overlap from time to time, but it's definitely way less than when we were 10 people.

Re: The Evolution of Code Deploys at Reddit

#16

Post by spladug »

mschuster91 wrote: post #15

A question out of curiosity: why did you choose to write your deployment tools from scratch, instead of going with something like Jenkins? And, how do/did you provision new servers? By hand, or did you use something like Chef/Puppet?

> why did you choose to write your deployment tools from scratch, instead of going with something like Jenkins?

Each step along the way was basically just a small modification on the system before. AFAIK Jenkins doesn't come with the ability to safely deploy code to hundreds of servers out of the box, so building out the systems to make that the case would've been more work than just adding to what existed and for unknown benefit.

> And, how do/did you provision new servers? By hand, or did you use something like Chef/Puppet?

We use puppet for configuration management. That's been the case for most stuff since early 2011 for context. There's a lot more detail in our semi-recent infra/ops AMA: https://www.reddit.com/r/sysadmin/comments/57ien6/were_reddi...

Re: The Evolution of Code Deploys at Reddit

#18

Post by spladug »

technion wrote: post #14

It also says a lot about how basic deployment schemes are perfectly good at a reasonable scale. I've seen many projects in the 1-2 server range become messy nightmares as people obsessed with 2017-reddit style deployments.

Keep it simple and boring as long as you can, and even then go with the boringest solution to your problems.

Re: The Evolution of Code Deploys at Reddit

#19

Post by siliconc0w »

It's nice to have a lot of options when managing deployments. Feature flags, canary deploys, A/B or green/blue, etc. Canaries are really nice to catch the majority of deploy issues (i.e 'it doesn't start up' errors, obvious exception spikes or performance regressions). Feature flags let you hand off control to individual product teams and encourage them to create 'bite sized' changes which can be flagged. Blue/green is also a nice way to reduce risk given more complicated cross-service changes where having another 'known good' copy of production around is helpful.

Re: The Evolution of Code Deploys at Reddit

#20

Post by syncopate »

spladug wrote: post #3
eriknstr wrote: post #2

I think it's strange how little attention this got here on HN. Submitted by one of the Reddit admins even. I'd ask you to do an AMA but I have no questions and besides you lot are pretty good at answering questions whenever they come up anyway so.

Thanks! Do feel free to ask if you think of something :)

When you deploy: Will you lose requests that are currently being processed while you restart the service? Also, will a server still receive requests while the new code is being started?
Post reply on HN