Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub incident: ongoing issues with Actions, Issues and other Git operations

githubstatus.com

Post reply on HN
Print view View original

121–130 of 130 posts

Re: GitHub incident: ongoing issues with Actions, Issues and other Git operations

#121

Post by bombolo »

grumple wrote: post #80
misnome wrote: post #58

Earlier quoted context omitted.

We have a self-hosted Premium instance and have 30min of downtime _every day_ while the database is frozen and backed up. We've been told that it's a known issue being discussed with GitLab but that could just be CYA. But in any case, it's the "at scale, while changing" that tends to cause problems. Perhaps this is a continuing argument for self-hosting, especially if you don't have to expose the instance publicly. B…

> We have a self-hosted Premium instance and have 30min of downtime _every day_ while the database is frozen and backed up. I'm confused. You can do zero-downtime backups and replication of databases. I don't know what your company / Gitlab are doing but it seems wrong.

And why aren't they doing it at 3am

Re: GitHub incident: ongoing issues with Actions, Issues and other Git operations

#122

Post by darkwater »

that_guy_iain wrote: post #115
darkwater wrote: post #96

Earlier quoted context omitted.

> while we have 3-4 days of downtime on our self hosted jenkins during the same time Wow. This sounds like a broken organization?

Sounds like a startup with no devops and before a lot of the tooling to make things better.

And why such a startup should install a Jenkins server instead of relying on some SaaS free-tier?

Re: GitHub incident: ongoing issues with Actions, Issues and other Git operations

#123

Post by that_guy_iain »

darkwater wrote: post #122
that_guy_iain wrote: post #115

Earlier quoted context omitted.

Sounds like a startup with no devops and before a lot of the tooling to make things better.

And why such a startup should install a Jenkins server instead of relying on some SaaS free-tier?

It was 10-years ago. There weren't really any SaaS with free tiers.

Re: GitHub incident: ongoing issues with Actions, Issues and other Git operations

#124

Post by dclowd9901 »

Can anyone from GH weigh in on this? We've had several major outages from GH over the last month or two, and the company has been completely silent on the causes, as well as any sort of remediation steps to fix stability.

As a somewhat large size org, we're now exploring other options for code hosting.

Re: GitHub incident: ongoing issues with Actions, Issues and other Git operations

#125

Post by RamblingCTO »

capableweb wrote: post #81
RamblingCTO wrote: post #73

Hm, we had this last week as well. Guess who had a demo yesterday, needed to deploy things on Thursday/Friday and couldn't :(

If this is your first time being affected, I guess you have a good reason to adjust your deployment structure so you can deploy manually if needed. If this is not the first time, hopefully at least it will be the last one :)

No worries, we can. But I like to rely on automation to spend my time on other stuff. Worked out good in the end though ;)

Re: GitHub incident: ongoing issues with Actions, Issues and other Git operations

#126

Post by RamblingCTO »

hnarn wrote: post #84
RamblingCTO wrote: post #73

Hm, we had this last week as well. Guess who had a demo yesterday, needed to deploy things on Thursday/Friday and couldn't :(

Maybe a good time to investigate whether it's possible to hold those demos without relying on third party services.

You're always relying on third parties. Always. Except if you run it locally. We're way beyond that. I deployed to production just fine. It's just a helper. It adds to the stress tho.

Re: GitHub incident: ongoing issues with Actions, Issues and other Git operations

#127

Post by [deleted] »

idk1 wrote: post #47

I debugged for one min at 11:59 trying to push, and then my eat-lunch notification came in at 12:00 and I opened hackernews with a tuna sandwich and this is super helpful because it means I wont need to debug it locally for 10 mins before figuring out Github is down. Edit - Just want to clarify when I say "opened hackernews with a tuna sandwich" I want to clear up that I did indeed full on mash the keyboard letters w…

[deleted]

Re: GitHub incident: ongoing issues with Actions, Issues and other Git operations

#128

Post by dinvlad »

NikxDa wrote: post #14

The GitHub Status shows 14 incidents affecting Git Operations this year alone [1]. That's quite a lot, considering it's only May. I wonder if the outages were always this frequent and just get more publicity on here now, or whether there was a significant increase in outages fairly recently. [1] https://www.githubstatus.com/history

I wonder if it has anything to do with layoffs, if crucial Ops people left

Re: GitHub incident: ongoing issues with Actions, Issues and other Git operations

#129

Post by lol768 »

ishanjain28 wrote: post #116
subtra3t wrote: post #75

Earlier quoted context omitted.

Just like calling GitHub "unreliable garbage"...

Github has been down hundreds of times this year alone. They have reported outages 72 times this year and there are multiple times when services are unavailable and they don't report it on the status page. I don't see how the two situations are comparable

> there are multiple times when services are unavailable and they don't report it on the status page.

There's no evidence that the exact same doesn't happen with GitLab. I've had it (consistently) 500 on me in the past when there's nothing on their status page to indicate any issues.

Re: GitHub incident: ongoing issues with Actions, Issues and other Git operations

#130

Post by ishanjain28 »

lol768 wrote: post #129
ishanjain28 wrote: post #116

Earlier quoted context omitted.

Github has been down hundreds of times this year alone. They have reported outages 72 times this year and there are multiple times when services are unavailable and they don't report it on the status page. I don't see how the two situations are comparable

> there are multiple times when services are unavailable and they don't report it on the status page. There's no evidence that the exact same doesn't happen with GitLab. I've had it (consistently) 500 on me in the past when there's nothing on their status page to indicate any issues.

Okay Agreed, Except

That's not the point of discussion. I didn't say Gitlab doesn't lie about it or heck, That it doesn't have worse uptime than Github.

My argument is that a company erasing 300GB production database once is not a stain on their competency and that it can not be compared to a company which has very frequent outages which also happens to lie when they have outages.

Post reply on HN