251–260 of 266 posts
> I wanted to give an update on GitHub’s availability in light of two recent incidents . [Emphasis mine] Vlad, you are living in a very different world to me. GitHub has suffered dozens and dozens of outages since the beginning of the year. It is notably less available and reliable than it was even as recently as last year. People have created dashboards and heatmaps showing how bad GitHub has become. At least one of…
Security or trust not even making the list.
anyone who's actually worked there, could you explain why they're finding scalability and reliability so hard? naively it seems like 'repo groups', ie clusters of repositories linked by being mutual forks, would be fairly isolated for the whole git storage layer, and everything else feels pretty easily parallelisable (issues, actions, etc, modulo taking locks now and then to submit results or whatever). and given tha…
What always struggled was the richness of the Rails monolith itself and its backing MySQL databases - the expectation that everything links to everything throughout the product (think: issue cross references across orgs that only appear if you're able to access the remote repo, and other things like that).
Those details appear richly everywhere you look, and the combination of that with a general lack of understanding and/or focus on performance (shipping big features is hard, shipping them with performance at scale is MUCH harder), compounded by Ruby being an easy language to get performance wrong in (object count really hurts, and it makes it very easy to create many) leaves every feature adding to the performance problem, and makes it daunting/impossible to make fast once it's slow.
There was a full on year or more of making GitHub fast while I was there that just couldn't gain enough momentum to make enough of a dent to make it better. I remember finding and fixing a N^3 (or maybe it was N^4? something bad) in the home page activity feed - the worst thing I found but gives an idea. IMO it would need a fresh view of how to keep interfaces simple and how to design the data layers performantly - not adding every bell and whistle to every screen.
I hope someone at GitHub realises they are about to lose everything that was hard earned by early GitHub - it once was a site people (myself included) looked up to for ideal availability, responsible releases, data driven improvement - but no more it seems :(
Earlier quoted context omitted.
It's entirely possible the move to Azure has made the availability problems worse. Dedicated hardware is much more predictable than cloud. "Let's not move to Azure and instead buy a few more racks" was likely a decision beyond the pay grade of github's management.
Moving to cloud makes scaling much easier and faster than colo data centers, though it cost more and might not be as reliable.
Earlier quoted context omitted.
As a business user, our costs have gone up while service has gone down dramatically. Meanwhile our marginal cost to GitHub has hardly changed. Where our costs to them have increased, they mostly charge us per cpu minute, so obviously aren’t making any kind of loss on our account. I’m sure they’re experiencing scaling issues across the platform, but it’s unacceptable for that to have a negative impact on us when we're…
> we're sending them $250/dev/yr for (what is in all honesty) hosting a bunch of static text files. You know, you can just host your own code forge. Or you can just drop gitolite on a server. Or pull directly from each others' dev machines on a LAN. GitHub is not git.
We're actively looking into alternatives outside of GitHub though.
Earlier quoted context omitted.
Why do we need to be charitable to Microsoft? Did we lose our ability to consider them the evil empire?
There’s a lot of “won’t someone think of the GitHub employees” on here
Many of those employees actually had stock options that exited at unicorn valuation. Must be nice.
Earlier quoted context omitted.
MAWS is “Move to AWS”, the name of the internal campaign to get legacy services into a somewhat-retrofitted AWS environment. It was a single VPC at one point.
I just finished a nearly five year stint at amazon and didn't realize there was pre-maws stuff still around. Never encountered any of it. I was like two months from my yellow badge but, uh, life is really better outside amazon.
Earlier quoted context omitted.
It's curious how bad people say Azure is. I've never used it, but I've used AWS, and AWS is a gigantic mess. So that makes me concerned if Azure is worse than a gigantic mass.
Azure's management APIs break connections coming from outside Azure's network every time they use DNS to execute a blue/green swap on their public load balancers. Existing connections are not gracefully drained. Terraform state gets corrupted (it thinks the operation failed when it actually succeeded and the resource was actually created) and requires manual fixing. This happened frequently enough at large enough sca…
It's quite incredible that a support bill in the >10k per month range from azure makes the public google (not even GCP with a support contract) support look not crap
Earlier quoted context omitted.
Seems pretty sensible to not rely on a single provider for their large complex system?
This isn't a mom and pop shop. They have locations all over the world: https://datacenters.microsoft.com/ There's no intrinsic reason they should be vulnerable to themselves.