Live data from Hacker News

The GitHub Load Balancer

githubengineering.com

71–80 of 137 posts

Re: The GitHub Load Balancer

#71

Earlier quoted context omitted.

Because if you say to management "hey, why don't we release the code that our entire business is built off of for anyone to host for themselves for free!" you'd be laughed out of the building on your way to the psych ward.

Their entire business is already built upon software that anyone can host themselves for free.

So github has no custom/proprietary code or value adding features?

Re: The GitHub Load Balancer

#72
I notice a lot of negativity arround here. Don't know why that is... But i'll take my 5 cents on it.

NIH - Not invented here and redoing an opensource project.

- Github said they used HAProxy before, i think the use case of github could very well be unique. So they created something that works best for them. They don't have to re-engineer an entire code base. When you work on small projects, you can send a merge request to do changes. I think this is something bigger then just a small bugfix ;). Totally understand them there for creating something new

- They used opensource based on number of open source projects including, haproxy, iptables, FoU and pf_ring. That is what opensource is, use opensource to create what suits you best. Every company has some edge cases. I have no doubt that Github has a lot of them ;)

Now,

Thanks GitHub for sharing, i'll follow up on your posts and hope to learn a couple of new things ;)

Re: The GitHub Load Balancer

#73
post #49

Earlier quoted context omitted.

here's an alternative interpretation: Due to their unique requirements, they were forced to investigate alternative approaches to the problem, and the approach they went with, they believe, is useful to a lot more people than just themselves. This is both plausible and matches the announcement, so why take a bad faith approach to someone releasing some code to the open source? Would you prefer if they kept it closed?…

Huh? Atom/Electron is another great example of GitHub duplicating a ton of existing projects (whether dependencies such as CEF and node-webkit or high-level solutions such as ACE) without seemingly having any interest at all in joining those existing projects. Just because someone is successful at doing this does not make what they are doing any more reasonable: if anything it should just put them in a similar place…

Or maybe they wanted to spend their energy actually solving their problems rather than trying to persuade the maintainers of other projects about their approach. For example, could they have proposed some changes to IPVS to get rid of multicast for state sharing? Maybe. But then they'd spend all that time arguing with other users of that project about the relative merits of each. Instead they built a new solution and users now can have a choice, including the choice to take some of these ideas and apply them to the other projects if they are clearly superior.

I would accuse them of NIH if they simply reinvented another wheel when there was a perfectly acceptable solution already out there. But it doesn't seem like that was the case. Instead they clearly evaluated the existing solutions, found shortcomings, and decided to solve those problems for themselves, and then publish the resulting code. I see nothing wrong with that approach.

Re: The GitHub Load Balancer

#74
post #69

They talk about running on "bare metal" but when I followed that link it looked like they were simply running under Ubuntu. Is it so much a given that everything is going to be virtualized? When I think of "bare metal" I think of a single image with disk management, network stack, and what few services they want all running in supervisory mode. Basically the architecture of an embedded system.

Yes, it is assumed that all startups are running in EC2 us-east-1 and "bare metal" is the accepted term for non-virtualized systems.

Re: The GitHub Load Balancer

#75
post #29

I am increasingly bothered by the "not invented here" syndrome where instead of taking existing projects and enhancing them, in true open source fashion, people instead re-create from scratch. It is then justified that their creation is needed because "no one else has these kinds of problems" but then they open source them as if lots of other people could benefit from it. Why open source something if it has an expect…

> in true open source fashion, people instead re-create from scratch.

True open source fashion is also the freedom to work on whatever you want.

"Just think about it, if they would have contributed to foo instead of working on bar, then foo would be twice as good!" keeps being thrown around every time someone announces something new around here.

Re: The GitHub Load Balancer

#76
post #40

Do the Directors use Anycast then? That wasn't clear to me.

Anycast usually implies traffic will be directed to the nearest node advertising that prefix. The GLB directors leverage ECMP which provides the ability to balance flows across many available paths.

Anycast and ECMP work together in the context of load balancing. ECMP without Anycasted destination IPs would be pointless for horizontally scaling your LB tier.

What Anycast means is just that multiple hosts share the same IP address - as opposed to unicast. When all the nodes sharing the same IP are on the same subnet "nearest" is kind of irrelevant. So the implication is different.

Re: The GitHub Load Balancer

#77
post #68
post #29

I am increasingly bothered by the "not invented here" syndrome where instead of taking existing projects and enhancing them, in true open source fashion, people instead re-create from scratch. It is then justified that their creation is needed because "no one else has these kinds of problems" but then they open source them as if lots of other people could benefit from it. Why open source something if it has an expect…

Have you ever contributed to HaProxy? Have you ever tried committing massive alterations to major open source projects? It isn't as simple as here's my massive rewrite, click the accept button and everything works out for the open source community. Let me be the first to say that the level of politics, circle jerking and knowing people is ridiculous.

Given the good reaction to an out-of-the-blue patch from me on the HAProxy mailing list, I'd imagine that contributing even major changes to HAProxy probably would go rather well. It's one of the best open source development communities I've experienced. Welcoming, but still highly focused on quality contributions. The quality and performance of HAProxy reflects this approach.

Re: The GitHub Load Balancer

#78

I love using GitHub and appreciate the impact it is and has had. But this post is what is wrong with the web today. They have taken a distributed-at-it's-plumbing technology, and centralised it so much that now we need to innovate new load balancing mechanisms. Years ago I worked at Demon Internet and we tried to give every dial up user a piece of webspace - just a disk always connected. Almost no one ever used them.…

You're not alone in thinking this (and I was one of those Demon customers back in the day :-) ). I've had pub chats with many tech folk over the years about how we might enable this. It needs to be cheap, of course, like 5 USD pa cheap. But, perhaps, with today's container tech that's viable. It needs other things too.

A fully distributed "web" of personally managed data is where we'll get to one day. It might need a few cycles of centralisation and distribution, though.

It's also one of the reasons why we must not let the "privacy is dead" and "back door encryption" folk have their way.

To be fair, though, I don't think GitHub is a big part of that problem.

Re: The GitHub Load Balancer

#79

I love using GitHub and appreciate the impact it is and has had. But this post is what is wrong with the web today. They have taken a distributed-at-it's-plumbing technology, and centralised it so much that now we need to innovate new load balancing mechanisms. Years ago I worked at Demon Internet and we tried to give every dial up user a piece of webspace - just a disk always connected. Almost no one ever used them.…

Are you saying instead of having Github, we should all be hosting our own Git repos? > But it is beginning to creak. GitHub faces scaling challenges, I don't agree that Github facing scaling issues means the web is creaking. More like old wooden boats are being replaced by big, sturdy battleships. I think the web is getting stronger thanks to engineers facing the challenges coming their way. > I am frustrated that so…

>>> Are you saying instead of having Github, we should all be hosting our own Git repos?

Well, yes. That's the point. It was designed as an entirely distributed setup. It's crazy that in order to post a message to my neighbours I have to send data to Facebook in SV and just as crazy that two devs on the same team need to write their code commits in a load balanced mega server in ... Err ... Washington? Wherever.

And I don't mind having lots of clients but I object to no open standards, incompatible and frequently unavailable APIs and lack of control over my messages and how they are dealt with. I want procmail for messaging platforms ! And I want a pony !

Re: The GitHub Load Balancer

#80

I love using GitHub and appreciate the impact it is and has had. But this post is what is wrong with the web today. They have taken a distributed-at-it's-plumbing technology, and centralised it so much that now we need to innovate new load balancing mechanisms. Years ago I worked at Demon Internet and we tried to give every dial up user a piece of webspace - just a disk always connected. Almost no one ever used them.…

I started coding 3 years ago, and only started playing around with VPSs a year ago. It's hard to explain how... unaware I was, that I could just have a little "plot of land" on the internet, not managed by anyone other than me (and DO).

This is very enlightening. One of things that everyone on HN think as obvious, yet is far from that to most folk; even those with a fair degree of technical knowledge.
Post reply on HN