Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was down

githubstatus.com

Post reply on HN
Print view View original

41–50 of 88 posts

Re: GitHub was down

#41

Post by stigz »

capableweb wrote: post #23
cs-szazz wrote: post #3

Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?

It's surprisingly easy, depending on your scale/scope of course. But in general, I've managed to build CI/CD pipelines that are tolerant of GitHub (or any service) failures by following these steps: 1. Use as little of the configuration language provided by the CI as possible (prefer shellscripts that you call in CI instead of having each step in a YAML config for example) 2. Make sure static content is in a Git repo…

Great advice, thank you.

Re: GitHub was down

#43

Post by titanomachy »

capableweb wrote: post #23
cs-szazz wrote: post #3

Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?

It's surprisingly easy, depending on your scale/scope of course. But in general, I've managed to build CI/CD pipelines that are tolerant of GitHub (or any service) failures by following these steps: 1. Use as little of the configuration language provided by the CI as possible (prefer shellscripts that you call in CI instead of having each step in a YAML config for example) 2. Make sure static content is in a Git repo…

What facility are you using for multi signature login?

Re: GitHub was down

#45

Post by tazjin »

turbonoobie wrote: post #17

This is becoming a regular occurrence by now.. I wonder if reliability has become less of a priority. As somebody with little to no experience of running things at scale I’m finding myself attributing this to some form of “move fast and break things”.

I remember something after their acquisition about new offers being lower than what certain people had previously, leading to important staff members leaving. This and some other issue I can't quite remember ... it was probably posted on HN :)

Re: GitHub was down

#46

Post by Florin_Andrei »

turbonoobie wrote: post #17

This is becoming a regular occurrence by now.. I wonder if reliability has become less of a priority. As somebody with little to no experience of running things at scale I’m finding myself attributing this to some form of “move fast and break things”.

> I’m finding myself attributing this to some form of “move fast and break things”

That was the case when they were the small and hungry startup.

Meanwhile they've been acquired by a giant corporation with a less than stellar reputation for reliability or quality. So it's most likely a case actually of "move slow and break things".

Re: GitHub was down

#47

Post by Comevius »

rklaehn wrote: post #12

Great opportunity to try out decentralised alternatives like https://radicle.xyz/

I noticed that Radicle claims similar functionality to centralized code collaboration platforms like GitHub, but Radicle itself is being developed on GitHub.

They could use some dogfooding, and new website.

Re: GitHub was down

#48

Post by drstewart »

brnt wrote: post #18

I was in the middle of some last minute pre-weekend PR review, and midway I discover it can't actually submit any of my comments. Is there a way to review and save (intermediate) state offline?

I believe Github saves your input to local browser storage, so just make the comments and don't submit them til later.

Re: GitHub was down

#49

Post by spondyl »

jborichevskiy wrote: post #34

I wonder if they track Github Status traffic volume as some sort of meta-indicator? Is it even viable? I was futzing around with the description for a PR and hitting save wouldn't update it, yet clicking edit would show the text I expected to see. Suspecting something was up I checked Github Status but it was green across the board. Assuming enough other people hit the same chain of events, could it provide a reliabl…

> I wonder if they track Github Status traffic volume as some sort of meta-indicator? Is it even viable?

Sure, the previous decent sized company (~1000+ devs) had that exact metric available.

Visits to the status page generally that is. Now whether you could actually correlate that to an increase in errors for a particular component, no so much ;)

I'm sure it's totally feasible but it requires a certain amount of discipline to have consistency in logging/metric standards across all your applications to some extent.

Even worse, some applications would return a shared error page but internally, I believe it was logged as a 301 redirect until someone spotted it :)

Re: GitHub was down

#50

Post by rvz »

cs-szazz wrote: post #3

Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?

Well unfortunately it seems everything I said 11 days ago has become a reality I'm afraid and I was still downvoted for pointing this truth out. [0]

Too many times I suggested everyone to begin self-hosting or have that as a backup but once again some think 'going all in on GitHub' is worth it. (It really is not the case)

[0] https://news.ycombinator.com/item?id=26301750

Post reply on HN