Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Git Operations Are Down

githubstatus.com

Post reply on HN
Print view View original

181–190 of 271 posts

Re: GitHub Git Operations Are Down

#181

Post by Gigachad »

0xbadcafebee wrote: post #112
parpfish wrote: post #100

Earlier quoted context omitted.

but if you self host, how do you anticipate power outages and natural disasters? you shift it from a problem of software reliability to a problem of physical infrastructure. at some point in the chain somebody has to do that, but i'd prefer the person doing that was somebody with DEEP experience in it that could give you some nice confident assurances.

Basically you rent space and machines in datacenters that are geographically distributed and designed to resist natural disasters

That's exactly what Github does already. I wouldn't bet on my own org being better at hosting github than github.

Re: GitHub Git Operations Are Down

#182

Post by outworlder »

jcims wrote: post #88

Tangential at best but I work in a regulated industry and we're starting to get some heavy scrutiny from external auditors and regulators on the way we plan to address extended outages and forced exits from third party vendors. This goes beyond basic DR/BCP and plans are reviewed with at least a high level scenario exercise. https://www.federalregister.gov/documents/2023/06/09/2023-12... On the surface a product like…

My company is replacing the entire system we built with Github actions. Our previous DR plan involved running our automation scripts to reprovision the infrastructure that manages the rest of the infrastructure (runs jenkins jobs, etc).

They are replacing everything with Github actions. I wonder what they are going to do when Github is down.

Re: GitHub Git Operations Are Down

#183

Post by globnomulous »

jcims wrote: post #88

Tangential at best but I work in a regulated industry and we're starting to get some heavy scrutiny from external auditors and regulators on the way we plan to address extended outages and forced exits from third party vendors. This goes beyond basic DR/BCP and plans are reviewed with at least a high level scenario exercise. https://www.federalregister.gov/documents/2023/06/09/2023-12... On the surface a product like…

I work on one part of a huge suite of interconneted services that have extremely strict SLAs for up-time. It really is an interesting problem, and the quantity of engineering resources devoted to ensuring availability and avoiding downtime -- basically fighting to make inherently unstable, complex systems stable and reliable -- is dumbfounding.

When I'm trying to explain to people what it's like to work on this kind of software, I like to use an analogy: it's as though I have my own personal brick, or group of bricks, in the great pyramids of Egypt, just a a tiny piece of a stupefyingly, inconceivably larger whole -- and when I twist my chisel and strike my block just so, at exactly the right (or rather the wrong) angle, I can shake the very foundations of Egypt.

Re: GitHub Git Operations Are Down

#184

Post by eru »

nine_k wrote: post #102
eru wrote: post #81

Earlier quoted context omitted.

> - Access policy enforcement (who can commit and when); Interestingly, what GitHub mostly enforces is where your branches point to. Not who can make commits. That's mostly because of how git works, not because of any grand design on GitHub's part.

It controls who can push commits to the main branch hosted by GitHub (and other branches if you want to configure that). You can have OWNERS files to control who can push commits touching particular parts of the tree, or who must approve such a push / merge (see "pull request"). Out of the box, git does not offer that, and this does require a single point of enforcement.

My point is that in git branches are just mutable pointers to commits. Tags are internally nearly the same, but socially they are meant to be immutable.

Anyone can make any commit they want in git. That includes merge commits, too. GitHub mostly lets anyone push any commits they feel like, too. (What restrictions are there on pushing commits is mostly to deal with denial of service and people being a nuisance.)

Where the policing comes in is in giving rules for how these pointers (aka branches) can be mutated. OWNERS files, PR reviews, CI automation etc is all about controlling that mutation.

See also the new-ish merge queues[0], which really bring out that difference: the merge queue machinery makes the merge commit of your approved PR branch with 'main', runs the CI against that, and iff that passes, moves the pointer that is 'main' to point to the newly created commit.

It's exactly the same commit (with exactly the same hash), whether it passes the CI or not. The only difference is in whether it gets the official blessing of being pointed to by the official 'main'.

It really speaks to the design of git, that conceptually the only thing they need to lock down is who can mutate this very small amount of data, these handfuls of pointers. Everything else is (conceptually) immutable, and thus you don't need to care about who can eg make commits.

[0] Really a re-implementation of bors-ng.

Re: GitHub Git Operations Are Down

#185

Post by eru »

rad_gruchalski wrote: post #108
eru wrote: post #75

Earlier quoted context omitted.

I actually don't care whether it's centralised or decentralised, or who's managing it. But you are right that I want reliable and easy-to-use services. And centralisation is often one way to go there. As an interesting counterpoint: Git itself is decentralised and replaced centralised services like Subversion. And that made git easier to use, especially easier to get started with: no need for a server, no need to be…

> no need for a server, no need to be online, just do `git init` in any old directory svn doesn’t require a server and there is no need to be online. It works perfectly fine over the file:// protocol.

Interesting.

Was that always the case? I remember it being quite a hassle to set up (following tutorials online), but that was about 15 to 20 years ago or so.

Re: GitHub Git Operations Are Down

#186

Post by eru »

umvi wrote: post #150
ascar wrote: post #131

Earlier quoted context omitted.

And a `git remote add name url` and you are setup to use another remote server.

Only if your repo doesn't have any other critical integrations like CI/CD, jira, etc

Why? We use github and its CI/CD system, but locally I still only need to add the git remote to work with it.

Re: GitHub Git Operations Are Down

#187

Post by flats »

sixo wrote: post #19
crest wrote: post #13

It's almost as if only Git is distributed, but people sold out to Github for convenience. Too bad Git lacks a distributed bug tracker and wiki system like Fossil. Guess Github has to fail a lot more for things to change.

people could not be more clear that their preference is for reliable and easy-to-use centralized services maintained by professionals, and not decentralized systems that require a great deal of user expertise

this should be on a wall somewhere

Re: GitHub Git Operations Are Down

#188

Post by Over2Chars »

hamandcheese wrote: post #180
Over2Chars wrote: post #175

Earlier quoted context omitted.

If all you care about is insurance, I think you can sometimes merely attest "yeah, it will work" and check a box. No demo necessary (sadly). If you actually care about uptime, then a real demo with usage is likely the better approach: switch over to your "backup" on a regular basis and make sure it works 100% as expected.

My hypothetical universe is "I believe GitHub is too big to fail and want to spend as little resources to please the auditor as is reasonably possible without resorting to fraud". So really what I'm asking is "how strict are these audits really?"

Github is owned by Microsoft so I'm assuming, maybe incorrectly, that they're well funded.

Internal audits are always subject to gaps, but if the stated issue is correct "a load balancer config change gone pear shaped" an audit wouldn't have caught that necessarily.

Unless the audit wants to test their change control, deployment methods, and redundancy.

Are they changing all of their load balancers all at once? Seems non optimal. Maybe change only one at a time, or a small batch.

Are they propagating load balancer changes from a canary to production without vetting it's good?

Or did they vet it and they were wrong - some difference in their canary or analysis had a short coming?

And even if all of that was A-OK why did a mistake (and we all make mistakes) not get reverted quickly?

Were there insufficient internal controls to revert small mistakes and keep them from becoming site wide outages? And so on.

I suspect these kinds of discussions are happening. Or, maybe not. Who knows?

It's a 3rd party, and even if your whole organization's life depends on it you only know what they tell you.

Welcome to "the cloud".

Re: GitHub Git Operations Are Down

#189

Post by thayne »

jcims wrote: post #88

Tangential at best but I work in a regulated industry and we're starting to get some heavy scrutiny from external auditors and regulators on the way we plan to address extended outages and forced exits from third party vendors. This goes beyond basic DR/BCP and plans are reviewed with at least a high level scenario exercise. https://www.federalregister.gov/documents/2023/06/09/2023-12... On the surface a product like…

Let's say you self-host Github. Now you are responsible for maintaining uptime, and you have less expertise with the service and less resources to dedicate to keeping it up, so it's going to be hard to match, much less exceed the uptime of Github cloud.

And it doesn't protect you from a "forced exit" either. Github could terminate your contract, and change the terms of the license in a way that you found unacceptable, or even go out of business, and being self hosted would leave you in no better position than if you had used cloud with external backups. You can somewhat mitigate this risk by self hosting an open source solution, so that in the worst case scenario, you can fork the project and maintain it yourself, but there is still risk that the project could be abandoned, or possibly have the license changed in future versions.

To be clear, I'm not saying that you shouldn't self host and SaaS is always better. But it isn't a magic bullet that solves these problems.

Re: GitHub Git Operations Are Down

#190

Post by bgrainger »

thayne wrote: post #189
jcims wrote: post #88

Tangential at best but I work in a regulated industry and we're starting to get some heavy scrutiny from external auditors and regulators on the way we plan to address extended outages and forced exits from third party vendors. This goes beyond basic DR/BCP and plans are reviewed with at least a high level scenario exercise. https://www.federalregister.gov/documents/2023/06/09/2023-12... On the surface a product like…

Let's say you self-host Github. Now you are responsible for maintaining uptime, and you have less expertise with the service and less resources to dedicate to keeping it up, so it's going to be hard to match, much less exceed the uptime of Github cloud. And it doesn't protect you from a "forced exit" either. Github could terminate your contract, and change the terms of the license in a way that you found unacceptable…

We self-host GitHub using GitHub Enterprise Server. It is a mature product that requires next-to-no maintenance and is remarkably stable. (We did have a period of downtime caused by running it on an underprovisioned VM for our needs, but since resolving that it hasn't had problems.)

Of course we have a small and mostly unchanging number of users, don't have to deal with DDoS attacks, and can schedule the fairly-infrequent updates during maintenance windows that are convenient for us (since we don't need 100% availability outside of US working hours).

I don't have the metrics in front of me, but I would say we've easily exceeded github.com's uptime in the last 12 months.

Post reply on HN