Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The GitHub Load Balancer

githubengineering.com

Post reply on HN
Print view View original

51–60 of 137 posts

Re: The GitHub Load Balancer

#51

Post by fps »

the_duke wrote: post #39
cortesoft wrote: post #13

Earlier quoted context omitted.

I think the model of "keep our core business software proprietary and open source all non-core software" is a good one, and one we should encourage all companies to share. If they open sourced their core site, how would they make any money?

Agreed. Considering the amazing value Github provides for the IT world, I really don't care if they release their code. Isn't it enought that they have revolutionized development workflows, project hosting and made contributing and collaborating so much easiert? (to be clear, GH wasn't the first site of it's kind, but the network effect of having one site that that basically every developer uses daily -and be it just…

I use github every day for professional and personal projects, and typically like what they've done, but Sourceforge.net was the github of the early 2000's, where almost every developer had an account and where you could get free development and distribution tools for your project. Look at what it is now (or was before it was re-opened a few months ago.) People's open source projects were being leveraged to push malware on unsuspecting users. We might not think it's so great to have all this centralized code storage in a few more years when the new hotness replaces github and everyone's zombie products are being used maliciously again.

Re: The GitHub Load Balancer

#52

Post by petercooper »

lifeisstillgood wrote: post #14

I love using GitHub and appreciate the impact it is and has had. But this post is what is wrong with the web today. They have taken a distributed-at-it's-plumbing technology, and centralised it so much that now we need to innovate new load balancing mechanisms. Years ago I worked at Demon Internet and we tried to give every dial up user a piece of webspace - just a disk always connected. Almost no one ever used them.…

Years ago I worked at Demon Internet and we tried to give every dial up user a piece of webspace - just a disk always connected. Almost no one ever used them.

I did. That was one of the best features of Demon at the time, when 10MB of Web space could cost you £100+ per year :-) Thanks for your part in making it work!

Re: The GitHub Load Balancer

#53

Post by jimjag »

logicalstack wrote: post #41
jimjag wrote: post #37

Earlier quoted context omitted.

They mention HAProxy, but it doesn't look like it's based on it.

Joe from GitHub here, we'll talk about it later posts but GLB is based on a number of open source projects including, haproxy, iptables, FoU and pf_ring. Many existing open source solutions are optimized for short lived HTTP requests and don't address the long running connection issue (like a large git clone). We wanted something better for our use case.

That is good to know. Thx.

Re: The GitHub Load Balancer

#54

Post by alexandros »

saurik wrote: post #49
alexandros wrote: post #42

Earlier quoted context omitted.

here's an alternative interpretation: Due to their unique requirements, they were forced to investigate alternative approaches to the problem, and the approach they went with, they believe, is useful to a lot more people than just themselves. This is both plausible and matches the announcement, so why take a bad faith approach to someone releasing some code to the open source? Would you prefer if they kept it closed?…

Huh? Atom/Electron is another great example of GitHub duplicating a ton of existing projects (whether dependencies such as CEF and node-webkit or high-level solutions such as ACE) without seemingly having any interest at all in joining those existing projects. Just because someone is successful at doing this does not make what they are doing any more reasonable: if anything it should just put them in a similar place…

Open source does not entail any responsibility to work or not work together with any pre-existing project. What you describe is more cathedral than bazaar. There are thousands of possible reasons, from architectural choices to personalities, that someone may have not chosen to work with an existing project. I will not fault them for giving me a superior result, for free (in both senses). To compare this with Microsoft in the nineties is sheer madness.

Re: The GitHub Load Balancer

#55

Post by seangrogg »

Ianvdl wrote: post #3

Given the title and the length of the post I was expecting a lot more detail. > Over the last year we’ve developed our new load balancer, called GLB (GitHub Load Balancer). Today, and over the next few weeks , we will be sharing the design and releasing its components as open source software. Is it common practice to do this? Most recent software/framework/service announcements I've read were just a single, longer po…

As a JS developer I see these all the time with respect to frameworks/libraries. Relay and GraphQL were both announced with high-level overviews long before being open-sourced. Angular 2 is another. It seems to be the case that such announcements are more to generate interest in the subject which then helps suss out requirements that may not have been considered by the developing team.

Re: The GitHub Load Balancer

#57

Post by olalonde »

alexandros wrote: post #42
jimjag wrote: post #29

I am increasingly bothered by the "not invented here" syndrome where instead of taking existing projects and enhancing them, in true open source fashion, people instead re-create from scratch. It is then justified that their creation is needed because "no one else has these kinds of problems" but then they open source them as if lots of other people could benefit from it. Why open source something if it has an expect…

here's an alternative interpretation: Due to their unique requirements, they were forced to investigate alternative approaches to the problem, and the approach they went with, they believe, is useful to a lot more people than just themselves. This is both plausible and matches the announcement, so why take a bad faith approach to someone releasing some code to the open source? Would you prefer if they kept it closed?…

> Due to their unique requirements [...] This is both plausible

I'm skeptical that Github's load balancing requirements are that different from any of the other large file hosts and SaaS companies. But it's possible and I'm not in a position to tell. That being said, the "NIH syndrome" is a largely overlooked problem in our industry and I think it's reasonable to raise concerns over new projects that may be reinventing the wheel.

Re: The GitHub Load Balancer

#58

Post by halifaxbeard »

logicalstack wrote: post #41
jimjag wrote: post #37

Earlier quoted context omitted.

They mention HAProxy, but it doesn't look like it's based on it.

Joe from GitHub here, we'll talk about it later posts but GLB is based on a number of open source projects including, haproxy, iptables, FoU and pf_ring. Many existing open source solutions are optimized for short lived HTTP requests and don't address the long running connection issue (like a large git clone). We wanted something better for our use case.

I'm currently working with GitHub Support on dealing with zip downloads of a 5GB repo failing after 2-3 minutes, with curl error "transfer closed with outstanding read data remaining".

Sure about the long running connection issue being solved? :-)

Re: The GitHub Load Balancer

#59

Post by nkassis »

tomschlick wrote: post #21
tadelle wrote: post #15

Earlier quoted context omitted.

But they could release trimmed community version at least ;)

Why? Open sourcing GH would probably be a huge undertaking for a considerable amount of their team (removing proprietary code, making sure no credentials are in there, etc).

Not to mention the fact that the whole of github is probably a bunch of disparate little project worked on by individual teams that would require a lot of effort to turn into something that could be run by an individual person. Probably makes sense to expose some of those projects on their own as open source project like they are doing here.
Post reply on HN