Live data from Hacker News

Scaling GitHub

zachholman.com

11–20 of 69 posts

Re: Scaling GitHub

#11
post #8

One thing the slides mention in passing that I cannot stress the usefulness of enough is an internal wiki. There are so many common tips, issues, ideas, etc. that can be more effectively communicated and expanded upon when they're in a centralized location for all to see. So many conversations of the form "I've never dealt with that, but I know $coworker ran into that once. Go ask him." can be reduced or eliminated i…

+1 But the issue I always face is that many coworkers aren't diligent enough to document in the wiki. How would you address that?

Re: Scaling GitHub

#12
post #8

One thing the slides mention in passing that I cannot stress the usefulness of enough is an internal wiki. There are so many common tips, issues, ideas, etc. that can be more effectively communicated and expanded upon when they're in a centralized location for all to see. So many conversations of the form "I've never dealt with that, but I know $coworker ran into that once. Go ask him." can be reduced or eliminated i…

+1 But the issue I always face is that many coworkers aren't diligent enough to document in the wiki. How would you address that?

Build that time into project budgets.

Re: Scaling GitHub

#13
post #9

Totally devoid of information. I would be more interested in reading actual technical reports of higly scaled infrastructures. These flimsy half-truths gleaned from such scaling, I can do without.

Agreed, but sadly this is the state of passing information now in "new age" startup/ companies. It's like Twitter: some nice pastel colors, very little text, no information but it's easy access and people can say "Oh yeah, I know about scaling I have read this on a blog post". Sorry to break it everyone but scaling is hard, totally unfunny or uncool, with tons of problems and stressful. But to answer your question an…

1) It's a slide deck for a talk. What were you expecting, a detailed tutorial? I also don't think you got the most important bit: focus on employees.

2) Scaling is also… really custom. I'm extremely unlikely to have to find a way to shard git repositories across a network of users. And they do cover their architecture and release back a fair number of their tools.

Re: Scaling GitHub

#14

I'm usually wary of Holman's presentations and writing. GitHub is such a unique, special place — an engineering company with a product engineers use and are devoted to. This presentation is pretty good and has some good tenets. However, I have trouble believing that all of these ideals will work when you have a PM screaming at your engineers to finish the internal CRM system, or something else that engineers must wor…

You're right, they wouldn't. GitHub would never hire the PM you're referring to, because that's the type of PM that makes engineers hate their jobs and become demotivated.

Re: Scaling GitHub

#15
post #9

Totally devoid of information. I would be more interested in reading actual technical reports of higly scaled infrastructures. These flimsy half-truths gleaned from such scaling, I can do without.

"Totally devoid of information."

That seems a little harsh for something that is clearly intended to be an overview of general principles, not specific technical details.

Re: Scaling GitHub

#16

I'm usually wary of Holman's presentations and writing. GitHub is such a unique, special place — an engineering company with a product engineers use and are devoted to. This presentation is pretty good and has some good tenets. However, I have trouble believing that all of these ideals will work when you have a PM screaming at your engineers to finish the internal CRM system, or something else that engineers must wor…

GH is a special case. They're darlings of the tech community, and they throw a ton of events.

Even if they were crappy to work for they would probably have an easy time hiring.

THAT SAID, "finishing the internal CRM system" is a problem that happens in organizations an order of magnitude larger, methinks.

My finishing platitude is, how did you hire - and why do you retain - engineers that don't give a shit about the project they're working on?

Also, why is your PM screaming? A project meeting deadlines is his or her responsibility. Sounds like you have a crap PM to boot.

Re: Scaling GitHub

#17
post #9

Totally devoid of information. I would be more interested in reading actual technical reports of higly scaled infrastructures. These flimsy half-truths gleaned from such scaling, I can do without.

I saw Zach Holman present this in Cape Town today. These slides are merely a back-drop to what was an excellent talk.

Re: Scaling GitHub

#19
post #9

Totally devoid of information. I would be more interested in reading actual technical reports of higly scaled infrastructures. These flimsy half-truths gleaned from such scaling, I can do without.

Agreed, but sadly this is the state of passing information now in "new age" startup/ companies. It's like Twitter: some nice pastel colors, very little text, no information but it's easy access and people can say "Oh yeah, I know about scaling I have read this on a blog post". Sorry to break it everyone but scaling is hard, totally unfunny or uncool, with tons of problems and stressful. But to answer your question an…

We've gone into excruciating detail about how GitHub's architected: https://github.com/blog/530-how-we-made-github-fast

Re: Scaling GitHub

#20
post #9

Totally devoid of information. I would be more interested in reading actual technical reports of higly scaled infrastructures. These flimsy half-truths gleaned from such scaling, I can do without.

Agreed, but sadly this is the state of passing information now in "new age" startup/ companies. It's like Twitter: some nice pastel colors, very little text, no information but it's easy access and people can say "Oh yeah, I know about scaling I have read this on a blog post". Sorry to break it everyone but scaling is hard, totally unfunny or uncool, with tons of problems and stressful. But to answer your question an…

Sorry to break it to you, but scaling isn't hard anymore. The tools we have at our disposal today makes it trivial compared to companies attempting to do the same thing 5-6 years ago. As soon as you figure out the scaling from 1 -> 2 servers, you're basically done until you hit top-1000 traffic. Because as soon as you spend a weekend doing that, scaling to 2 -> N is child's play.

Oh look, I'll pick from any number of battle-tested, high performance, and free load balancers. I'll spin up a couple more VPS nodes. I'll add a few more slaves to my DB setup. I'll install some memcached and a quick write-through wrapper in my code. Oh yeah, duh, don't do long-running operations on the request/response cycle, here's an open source queue system for you. And since I like going overboard, let's throw some Varnish all over the place. Congrats, we've just covered the scaling concerns of 95% of web apps.

Try doing that several years ago with the terrible PHP and Java services we were writing, and without most of the hardened tools we now get from thirty seconds of Googling and learning from everyone else's mistakes.

Disclaimer: yes, when you start to break through the top-1000 some hard problems start appearing again. You begin to run into the limits of what others have created, and patching / rolling your own solutions becomes commonplace.

Post reply on HN