When GitHub is down we legally don't have to do any work, it's in the constitution I swear
Best comment in this thread by far
GitHub degraded performance – resolved
31–40 of 58 posts
Re: GitHub degraded performance – resolved
#32Earlier quoted context omitted.
> It's basically a lost day of productivity even if GH goes down for only 30 minutes. Yes, we can continue coding locally, but issues & PRs are a huge part of our daily process. I know outages are frustrating, but how does 30 minutes before 10am ruin an entire day? Maybe you’re just being hyperbolic, but people take coffee breaks longer than that.
> how does 30 minutes before 10am ruin an entire day? Not everyone is on pacific time. This impacted us right in the middle of a standup call and disrupted our planning for the day. Also, the problem is more that you don't know how long the outage is going to last at first, so you start finding other ways to occupy your time. Through the lens of hindsight, yes we are certainly being hyperbolic in those cases where it…
> so you start finding other ways to occupy your time
Ok, pick a ticket and do some work on it locally, when that's done do the same with another. I can go a full day without interacting with github because I'm working on a local branch. Make a note of what branches you need to push later. I can't possibly imagine throwing my arms up and saying the day is wasted because I have to work locally. It's completely unbelievable.
Re: GitHub degraded performance – resolved
#33Earlier quoted context omitted.
> how does 30 minutes before 10am ruin an entire day? Not everyone is on pacific time. This impacted us right in the middle of a standup call and disrupted our planning for the day. Also, the problem is more that you don't know how long the outage is going to last at first, so you start finding other ways to occupy your time. Through the lens of hindsight, yes we are certainly being hyperbolic in those cases where it…
> so you start finding other ways to occupy your time Ok, pick a ticket and do some work on it locally, when that's done do the same with another. I can go a full day without interacting with github because I'm working on a local branch. Make a note of what branches you need to push later. I can't possibly imagine throwing my arms up and saying the day is wasted because I have to work locally. It's completely unbelie…
I understand your point and agree, but "pick a ticket" can be hard if you use GitHub issues for those.
Re: GitHub degraded performance – resolved
#34Hmm, how do I get the Github Actions CI to run on all the already existing PR's for which it never ran? Anyone know?
Re: GitHub degraded performance – resolved
#35This one is pretty nasty. Getting tired of the disengagement this causes for the team. It's basically a lost day of productivity even if GH goes down for only 30 minutes. Yes, we can continue coding locally, but issues & PRs are a huge part of our daily process. When I get back from vacation we are moving our shit to the enterprise plan. $21/user-month is really not that big of a deal when you are running basically y…
> It's basically a lost day of productivity even if GH goes down for only 30 minutes. Are we entering an era where if we don't have hundreds of thousands of servers running 24/7 to host our services, with all the resource consumption and environmental implications that result, that we will no longer be able to remain productive as a society? Is this gradually becoming a new baseline for humanity from which we cannot…
Yes. You're just noticing?
Re: GitHub degraded performance – resolved
#36This incident has been marked as resolved. You guys could check how often Github notable incidents that really captures the audience's attention in HN [0] [1]: [0] https://news.ycombinator.com/from?site=githubstatus.com [1] https://hn.algolia.com/?dateRange=pastYear&page=0&prefix=fal...
Looks like about once every two months?
Which is actually more than I expected, and seems like kind of too much.
Re: GitHub degraded performance – resolved
#37This one is pretty nasty. Getting tired of the disengagement this causes for the team. It's basically a lost day of productivity even if GH goes down for only 30 minutes. Yes, we can continue coding locally, but issues & PRs are a huge part of our daily process. When I get back from vacation we are moving our shit to the enterprise plan. $21/user-month is really not that big of a deal when you are running basically y…
> It's basically a lost day of productivity even if GH goes down for only 30 minutes. Are we entering an era where if we don't have hundreds of thousands of servers running 24/7 to host our services, with all the resource consumption and environmental implications that result, that we will no longer be able to remain productive as a society? Is this gradually becoming a new baseline for humanity from which we cannot…
Yes.
It's worth noting that you can take almost any software from before the late 90s to early 2000s, depending on the vendor, that is still available, and with a layer of emulation get it running in minutes.
The vast majority of software that is being built today for end users simply will not function in a short time frame because of aggressively built in dependencies on cloud based services, often with those dependencies designed to encourage customer lock-in and prevent piracy by forcing users to have active accounts and shift core logic from endpoints to cloud services.
Even moving past licensing servers and account capabilities, tools like Grammarly ship much of their analysis to the cloud, same for most translation services. Many modern text to speech services are cloud based as well (just look at how useless a modern cell phone becomes when you are without a data connection, for example).
I don't know what the statistics would look like, but I shudder to think how much of the world economy would grind to a halt if Amazon or another significant cloud provider had a sustained, multi-region outage (say 24-48 hours).
It's a god-damn mess, and we did it to ourselves.
Re: GitHub degraded performance – resolved
#38This one is pretty nasty. Getting tired of the disengagement this causes for the team. It's basically a lost day of productivity even if GH goes down for only 30 minutes. Yes, we can continue coding locally, but issues & PRs are a huge part of our daily process. When I get back from vacation we are moving our shit to the enterprise plan. $21/user-month is really not that big of a deal when you are running basically y…
On the plus side, you have a process and in theory it works, or you would be looking elsewhere.
You might have lost a day today, but how many days have you gained thanks to these tools the last month?
Re: GitHub degraded performance – resolved
#39Earlier quoted context omitted.
> It's basically a lost day of productivity even if GH goes down for only 30 minutes. Are we entering an era where if we don't have hundreds of thousands of servers running 24/7 to host our services, with all the resource consumption and environmental implications that result, that we will no longer be able to remain productive as a society? Is this gradually becoming a new baseline for humanity from which we cannot…
Yes. It's worth noting that you can take almost any software from before the late 90s to early 2000s, depending on the vendor, that is still available, and with a layer of emulation get it running in minutes. The vast majority of software that is being built today for end users simply will not function in a short time frame because of aggressively built in dependencies on cloud based services, often with those depend…
> I shudder to think how much of the world economy would grind to a halt if Amazon or another significant cloud provider had a sustained, multi-region outage (say 24-48 hours).
You can rest at ease. Nothing that is mission-critical for the world's financial infrastructure is hosted within one of these sorts of facilities. Facebook and Netflix might go down for days, but your Amex will still work at any merchant with a functioning internet connection.
I have been inside of [financial services organization]'s datacenter in [some midwestern state] which was purpose-built for the IT load. The strategy is "failure is not an option". It's essentially 1 gigantic, redundant life-support system for the one of the more sensitive computers on the planet. Amazon and Microsoft cannot afford to go to these lengths for the market they serve.
Re: GitHub degraded performance – resolved
#40This one is pretty nasty. Getting tired of the disengagement this causes for the team. It's basically a lost day of productivity even if GH goes down for only 30 minutes. Yes, we can continue coding locally, but issues & PRs are a huge part of our daily process. When I get back from vacation we are moving our shit to the enterprise plan. $21/user-month is really not that big of a deal when you are running basically y…
> There is an Azure datacenter within 100 miles of many of our home offices and I can ensure that our Github stack spins up there. Minimizing the amount of internet you have to transit to get to your applications can sidestep a lot of this stormy public cloud/internet weather bullshit. You've no guarantees that your local'ish data centre is going to be hop-wise, route-wise and peering-wise any better than a DC 1500 m…
> You've no guarantees that your local'ish data centre is going to be hop-wise, route-wise and peering-wise any better than a DC 1500 miles away, in relation to your home or office ISP.
You're correct. In fact, as I type this reply my cloudflare diagnostics are indicating I am talking to a datacenter 200 miles further away than would otherwise be ideal. That said, its still within an extremely reasonable distance. This is a "risk" I am willing to take. It's certainly a better starting point than guaranteed 70ms minimums.