Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub was down

githubstatus.com

Post reply on HN
Print view View original

61–70 of 88 posts

Re: GitHub was down

#61

Post by koluna »

jtdev wrote: post #31
qbasic_forever wrote: post #28

Github folks--this is really getting bad. I find it strange that your leadership will spends weeks of time, and pen hundreds of words about making right the wrongs they created with censorship (see: https://github.blog/2020-11-16-standing-up-for-developers-yo... ), yet there's almost no attention given to these major outages that keep happening for a year now. Where is the acknowledgment of a problem, root-cause anal…

Critical issue: "change master to main across the board"

You might be surprised to learn that there are many teams responsible for many things, and SRE is very likely a team different than the one that handles user experience decisions

Re: GitHub was down

#62

Post by Uehreka »

keithba wrote: post #60

Hi all - I'm the head of engineering at GitHub. Please accept my sincere apology for this downtime. The cause was a bad deploy (a db migration that changed an index). We were able to revert in about 30 minutes. This is slower than we'd like, and we'll be doing a full RCA of this outage. For those who are interested, on the first Wednesday of each month, I write a blog post on our availability. Most recent one is here…

I was just griping on Twitter yesterday about how many developers won't immediately revert an update that causes downtime, but will actually spend time trying to solve the problem while Rome burns.

Thank you for not doing that.

Re: GitHub was down

#63

Post by capableweb »

rvz wrote: post #50
cs-szazz wrote: post #3

Unfortunately right when we were trying to deploy a hot-fix to production, our CI can't clone the PR to run tests. What do other folks use to avoid this situation? Have a Gitlab instance or similar that you can pull from instead for CI?

Well unfortunately it seems everything I said 11 days ago has become a reality I'm afraid and I was still downvoted for pointing this truth out. [0] Too many times I suggested everyone to begin self-hosting or have that as a backup but once again some think 'going all in on GitHub' is worth it. (It really is not the case) [0] https://news.ycombinator.com/item?id=26301750

Something I've learned on HN is that upvotes/downvotes means nothing but how popular an opinion is. You can be 100% right, honest, straightforward and kind, but if the hive-mind does not agree, it does not agree and will downvote your well-written opinion.

Don't read too much into it and comment freely as normal. In the end, it's just internet points.

Re: GitHub was down

#64

Post by mwcampbell »

keithba wrote: post #60

Hi all - I'm the head of engineering at GitHub. Please accept my sincere apology for this downtime. The cause was a bad deploy (a db migration that changed an index). We were able to revert in about 30 minutes. This is slower than we'd like, and we'll be doing a full RCA of this outage. For those who are interested, on the first Wednesday of each month, I write a blog post on our availability. Most recent one is here…

Thanks for taking the time to personally give a status update while things are on fire. I hope you and all the others who are dealing with this emergency will have an especially restful weekend.

Re: GitHub was down

#65

Post by hanniabu »

keithba wrote: post #60

Hi all - I'm the head of engineering at GitHub. Please accept my sincere apology for this downtime. The cause was a bad deploy (a db migration that changed an index). We were able to revert in about 30 minutes. This is slower than we'd like, and we'll be doing a full RCA of this outage. For those who are interested, on the first Wednesday of each month, I write a blog post on our availability. Most recent one is here…

Any comment or insight you can share on the overall increase in downtime over the past few years?

Re: GitHub was down

#66

Post by bombcar »

usui wrote: post #53
whoisjohnkid wrote: post #29

Earlier quoted context omitted.

good point; I think they might be deploying big features on fridays? as of late I've noticed most of the issues I have seen happen on Fridays.

Why would that be the case? Shouldn't it be more common to find deployments happening early Monday morning? It's common practice to avoid potentially bug-inducing changes right before the weekend hits

If you’re trying to minimize impact to employees you deploy Monday or Tuesday morning.

If you’re trying to minimize impact to business customers you deploy Friday night.

Re: GitHub was down

#67

Post by jtdev »

koluna wrote: post #61
jtdev wrote: post #31

Earlier quoted context omitted.

Critical issue: "change master to main across the board"

You might be surprised to learn that there are many teams responsible for many things, and SRE is very likely a team different than the one that handles user experience decisions

They should have went to the mat over that ridiculous, pc, woke bullshit decision.

Re: GitHub was down

#68

Post by SemiNormal »

usui wrote: post #53
whoisjohnkid wrote: post #29

Earlier quoted context omitted.

good point; I think they might be deploying big features on fridays? as of late I've noticed most of the issues I have seen happen on Fridays.

Why would that be the case? Shouldn't it be more common to find deployments happening early Monday morning? It's common practice to avoid potentially bug-inducing changes right before the weekend hits

Because they likely get a large number of users doing git pull first thing Monday morning.

Re: GitHub was down

#69

Post by nix23 »

Florin_Andrei wrote: post #46
turbonoobie wrote: post #17

This is becoming a regular occurrence by now.. I wonder if reliability has become less of a priority. As somebody with little to no experience of running things at scale I’m finding myself attributing this to some form of “move fast and break things”.

> I’m finding myself attributing this to some form of “move fast and break things” That was the case when they were the small and hungry startup. Meanwhile they've been acquired by a giant corporation with a less than stellar reputation for reliability or quality. So it's most likely a case actually of "move slow and break things".

>stellar reputation for reliability or quality

That is unfair Minesweeper never crashed, and the printspooler is not up for debate here ;)

Re: GitHub was down

#70

Post by geerlingguy »

turbonoobie wrote: post #17

This is becoming a regular occurrence by now.. I wonder if reliability has become less of a priority. As somebody with little to no experience of running things at scale I’m finding myself attributing this to some form of “move fast and break things”.

Nobody remembers the unicorn days? Earlier in GitHub's history, it seemed like a weekly outage was the norm. You just kind of expected it and built workflows in ways where you had a backup path to your code.

Often git operations were unaffected though.

Post reply on HN