Prob just a coincidence, but our Memorystore (hosted redis) instance went down with a "repairing" status around the same time.
Gitlab is down
41–50 of 65 posts
Re: Gitlab is down
#42Earlier quoted context omitted.
It doesn't matter if each individual detects the outage because they'll start blame at the local source and move further up the tree rather than assign blame to a full system failure right off the bat. 99.9% of the time it's going to be a local failure affecting the individual. Also, most alerting systems like check multiple times before declaring a public outage, many times 2 to 3 failures some seconds apart are nee…
To add onto this, my experience is you never want a fully automated status page for another two reasons: 1. External engineers will start to automate recovery/mitigation processes around your status page if it has real time status. 2. You now need to bug test your status page thoroughly because of #1. It basically becomes an actual API.
That sounds like a problem for the external engineers, not for GitLab.
Re: Gitlab is down
#43overleaf.com is also down, https://status.overleaf.com/ Maybe some common severs ?
Definitely related, overleaf probably depends on gitlab. The overleaf outage ended right when the gitlab outage did.
Re: Gitlab is down
#44Earlier quoted context omitted.
Your impression is correct. We use GitLab.com and notice these incidents as they happen. The delay in updating status is a result of our Incident Management process [0]. We have a Communications Manager on Call (CMOC) who leads communication throughout an incident. One of their responsibilities includes updating the status page. The slight delay between noticing the issue and updating the status page is a result of t…
Not a "status page" then, but merely "a page where Communications Manager post messages on after assessing the situation and consulting/getting permission from management"
[deleted]
Re: Gitlab is down
#45Earlier quoted context omitted.
It took (by my measure) 13 minutes for a full outage to be represented on the status page. I was under the impression that gitlab use gitlab.com for their work. Surely someone would have noticed within seconds that it was down? Why have the misleading "updated a few seconds" ago text if it doesn't update on complete failure? :)
Your impression is correct. We use GitLab.com and notice these incidents as they happen. The delay in updating status is a result of our Incident Management process [0]. We have a Communications Manager on Call (CMOC) who leads communication throughout an incident. One of their responsibilities includes updating the status page. The slight delay between noticing the issue and updating the status page is a result of t…
Why is "Active Incident" and "System Wide Outage" on the status page with a background color of green? Why not red?
At first glance it looks like everything is operational with no issues.
Re: Gitlab is down
#46And this is why you self-host on your own instance.
Indeed, I can't remember a single time where a self-hosted server crashed. They run for decades with 0 downtime.
It doesn't, but I can fix it as opposed to waiting for their team to do it.
Also at gitlab.com scale the problems they face are very different from a typical deployment.
It is like having maintaining your car and using the train.
On average if you can fix your car (or hire a good mechanic i.e. consulting) you would probably have a better experience than public transport breaking down, that you are powerless to do anything about.
I would rather run a business depending on my car than the train ?
Re: Gitlab is down
#47Earlier quoted context omitted.
To add onto this, my experience is you never want a fully automated status page for another two reasons: 1. External engineers will start to automate recovery/mitigation processes around your status page if it has real time status. 2. You now need to bug test your status page thoroughly because of #1. It basically becomes an actual API.
That sounds like a problem for the external engineers, not for GitLab.
Users make their problems yours.
Re: Gitlab is down
#48Earlier quoted context omitted.
Indeed, I can't remember a single time where a self-hosted server crashed. They run for decades with 0 downtime.
And even if it goes down you might have more options to get it back to work.
I mean.. you can still use your git repos.
Need to do a launch? Build it and push it.
Need to share a change with someone so they can review?, `git diff` and send a patch via email. Want to use a server? Spin up a server, add users and keys and push up to it.
Gitlab, GitHub and these hosted solutions haven't always existed. They're convenient, but not a OMGWTF moment... unless of course you don't have backups.
Re: Gitlab is down
#49Earlier quoted context omitted.
Your impression is correct. We use GitLab.com and notice these incidents as they happen. The delay in updating status is a result of our Incident Management process [0]. We have a Communications Manager on Call (CMOC) who leads communication throughout an incident. One of their responsibilities includes updating the status page. The slight delay between noticing the issue and updating the status page is a result of t…
Why is "Active Incident" and "System Wide Outage" on the status page with a background color of green? Why not red? At first glance it looks like everything is operational with no issues.
I noticed that too. Pretty confusing to read "Operational" with a green background and "System Wide Outage" on the left side.
Re: Gitlab is down
#50Earlier quoted context omitted.
Indeed, I can't remember a single time where a self-hosted server crashed. They run for decades with 0 downtime.
Exactly. That is the whole point. I keep telling that for GitHub since that goes down once a month. [0][1] GitLab SaSS is the same but a self-hosted backup is better. [0] https://news.ycombinator.com/item?id=29901564 [1] https://news.ycombinator.com/item?id=29379648
Everyone should have backups and these things aren't infallible.
Gitlab is a perfect example. They had database issues and had to restore from backups already.