Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub: October 21 Incident Report

blog.github.com

Post reply on HN
Print view View original

51–60 of 119 posts

Re: GitHub: October 21 Incident Report

#51

Post by dlahoda »

metaphor wrote: post #50
lugg wrote: post #30

Earlier quoted context omitted.

> Define regularly More often than what is considered a standard 99.99% uptime SLA? (about an hour per year.) You seem to be making it out like a couple of days a year of lost [1] developer productivity is no big deal. That said, these things happen and you should probably check your workflows if you're all that blocked by GitHub being down.

> More often than what is considered a standard 99.99% uptime SLA? GitHub SLA is 99.95% and apparently exclusive to Business Cloud customers[1]. [1] https://github.com/pricing

It seems kind of 5 hours per year.

Re: GitHub: October 21 Incident Report

#52

Post by nik736 »

laumars wrote: post #14
nik736 wrote: post #6

I don't understand why code hosting platforms like GitHub, GitLab or BitBucket have so many issues regularly. Is there anything special about it?

> Is there anything special about it? Yup. You notice when they're down. Whereas when your self-hosted git server goes offline for a couple of hours, nobody else notices.

I am not talking about git hosting per se, but compared to other SaaS companies it seems GitHub/GitLab and Bitbucket are down much more often.

Re: GitHub: October 21 Incident Report

#53

Post by Klathmon »

_verandaguy wrote: post #49

I saw some comments on reddit which highlighted a pretty serious problem - many orgs rely on github as a fully integrated CD platform, with everything from code hosting, to running CI hooks, to pushing to staging or prod. It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over, but which has regularly been targeted in attacks b…

It's a tradeoff, just like anything else.

We use Github heavily at my work, and at past jobs as well. But at the same time it's not the ONLY way we can work. If github has an outage, our CD will shut down, but we can still run all tests locally, and we can still push to the server directly, and we can still push code between ourselves manually and review it.

Sure, it's a hiccup in our day when it goes down, but it's not like the entire company grinds to a halt. And the alternative of maintaining servers and systems to replicate all of that would take significantly more time, potentially cost more, and is probably more likely to go down.

Re: GitHub: October 21 Incident Report

#54

Post by hoppelhase »

foxhop wrote: post #43

Don't try creating a new repo, it will create part of the meta data but not allow you to see or use the repo and the repo name gets taken.

I created a new repository 30 minutes ago.

As I initialized it with a readme and an ignore file, I had to clone it. Cloning only succeeded by doing `watch git clone` and waiting a few minutes. But it worked.

Re: GitHub: October 21 Incident Report

#55

Post by geerlingguy »

rashthedude wrote: post #8
vqng wrote: post #5

> During this time, information displayed on GitHub.com is likely to appear out of date; if the new data is not presented, users will typically retry which may result in duplicated new content.

Happened to me already. Created a new branch but it kept me informing me the non-existence, retried a few times and shortly after was created with a bunch of duplicates.

Same here, but last night I was trying to post a comment to another repo, refreshed and tried again like 5 times before realizing there was an outage and it wasn't on my end. This morning that issue has 5 dupe comments (so I look like an idiot), and deleting them does nothing; they just reappear when I refresh the issue.

Re: GitHub: October 21 Incident Report

#56

Post by colonelpopcorn »

Klathmon wrote: post #53
_verandaguy wrote: post #49

I saw some comments on reddit which highlighted a pretty serious problem - many orgs rely on github as a fully integrated CD platform, with everything from code hosting, to running CI hooks, to pushing to staging or prod. It seems very unwise to have essentially your whole deployment process manager in the hands of an entity which you don't only have no control over, but which has regularly been targeted in attacks b…

It's a tradeoff, just like anything else. We use Github heavily at my work, and at past jobs as well. But at the same time it's not the ONLY way we can work. If github has an outage, our CD will shut down, but we can still run all tests locally, and we can still push to the server directly, and we can still push code between ourselves manually and review it. Sure, it's a hiccup in our day when it goes down, but it's…

On premises git hosting with Gittea or GitLab with mirrors to GitHub seems like a smart idea going forward.

Re: GitHub: October 21 Incident Report

#57

Post by elliotlarson »

colonelpopcorn wrote: post #56
Klathmon wrote: post #53

Earlier quoted context omitted.

It's a tradeoff, just like anything else. We use Github heavily at my work, and at past jobs as well. But at the same time it's not the ONLY way we can work. If github has an outage, our CD will shut down, but we can still run all tests locally, and we can still push to the server directly, and we can still push code between ourselves manually and review it. Sure, it's a hiccup in our day when it goes down, but it's…

On premises git hosting with Gittea or GitLab with mirrors to GitHub seems like a smart idea going forward.

But then you have to host it and maintain it. It's a slippery slope. How many 3rd party services do you in-house with hosted OS software. Pretty soon you're spending a huge chunk of your time doing ops work. And, where do you host it? On AWS, which can also go down, or on hardware hosted at your office. With on premise hosting, now you're in the hardware game too.

Re: GitHub: October 21 Incident Report

#58

Post by [deleted] »

colonelpopcorn wrote: post #56
Klathmon wrote: post #53

Earlier quoted context omitted.

It's a tradeoff, just like anything else. We use Github heavily at my work, and at past jobs as well. But at the same time it's not the ONLY way we can work. If github has an outage, our CD will shut down, but we can still run all tests locally, and we can still push to the server directly, and we can still push code between ourselves manually and review it. Sure, it's a hiccup in our day when it goes down, but it's…

On premises git hosting with Gittea or GitLab with mirrors to GitHub seems like a smart idea going forward.

[deleted]

Re: GitHub: October 21 Incident Report

#59

Post by Piskvorrr »

LeoNatan25 wrote: post #28
lol768 wrote: post #27

Earlier quoted context omitted.

I see an update 7 minutes ago. >12:56 British Summer Time >The majority of restore processes have completed. We anticipate all data stores will be fully consistent within the next hour.

Every hour they promise something will be done until the next hour. I haven’t been able to work all day so far.

Consider this a lesson on serverlessness. (We have been similarly afflicted, but their git backend seems to be up; and even further, we have rediscovered what we stopped paying attention to: that with Git, a centralized repo is just a convenience, not a requirement.)

Re: GitHub: October 21 Incident Report

#60

Post by LeoNatan25 »

Piskvorrr wrote: post #59
LeoNatan25 wrote: post #28

Earlier quoted context omitted.

Every hour they promise something will be done until the next hour. I haven’t been able to work all day so far.

Consider this a lesson on serverlessness. (We have been similarly afflicted, but their git backend seems to be up; and even further, we have rediscovered what we stopped paying attention to: that with Git, a centralized repo is just a convenience, not a requirement.)

Yes, I agree, but here it’s not up to me to choose the infrastructure.

I wonder what the total cost of this ordeal must be. Surely in the tens of millions.

Post reply on HN