It's interesting that different pieces of gitlab.com appear to be running on a hodge-podge of GCP, DO, AWS and AZ... I wonder why that would be the case?
But but CLOUD NATIVE! https://about.gitlab.com/cloud-native/
Gitlab is down
31–40 of 65 posts
Re: Gitlab is down
#32And this is why you self-host on your own instance.
Indeed, I can't remember a single time where a self-hosted server crashed. They run for decades with 0 downtime.
Well, my GitLab instance at some point started to have its Prometheus eat 100% CPU all the time until I disabled the Prometheus component altogether, so there’s that. A cursory glance at the tracker just now says the issue is still open. That’s the kind of problems you get for self-hosting, it’s not all rainbows and unicorns.
Re: Gitlab is down
#33GitLab team member here. We're aware of the incident and the status page has been updated. We will provide further updates on the status page as they become available. (Edited now that the status page has been updated).
It took (by my measure) 13 minutes for a full outage to be represented on the status page. I was under the impression that gitlab use gitlab.com for their work. Surely someone would have noticed within seconds that it was down? Why have the misleading "updated a few seconds" ago text if it doesn't update on complete failure? :)
Your impression is correct. We use GitLab.com and notice these incidents as they happen.
The delay in updating status is a result of our Incident Management process [0]. We have a Communications Manager on Call (CMOC) who leads communication throughout an incident. One of their responsibilities includes updating the status page. The slight delay between noticing the issue and updating the status page is a result of the time it takes for the CMOC to get alerted, assess the situation, and write the communication that is shared on the status page.
I'm not sure how the "updated a few seconds ago" messages are generated but I'll try to find out once the incident has been resolved.
0 - https://about.gitlab.com/handbook/engineering/infrastructure...
Re: Gitlab is down
#34Earlier quoted context omitted.
It took (by my measure) 13 minutes for a full outage to be represented on the status page. I was under the impression that gitlab use gitlab.com for their work. Surely someone would have noticed within seconds that it was down? Why have the misleading "updated a few seconds" ago text if it doesn't update on complete failure? :)
It doesn't matter if each individual detects the outage because they'll start blame at the local source and move further up the tree rather than assign blame to a full system failure right off the bat. 99.9% of the time it's going to be a local failure affecting the individual. Also, most alerting systems like check multiple times before declaring a public outage, many times 2 to 3 failures some seconds apart are nee…
To add onto this, my experience is you never want a fully automated status page for another two reasons:
1. External engineers will start to automate recovery/mitigation processes around your status page if it has real time status.
2. You now need to bug test your status page thoroughly because of #1. It basically becomes an actual API.
Re: Gitlab is down
#35Re: Gitlab is down
#36For SaSS, it is down. But not if you are self-hosting your own. Just look at Gnome: [0]. They are doing it right. [0] https://git.gnome.org
Is gitlab.gnome.org/GNOME set to forward to git.gnome.org?
Re: Gitlab is down
#37Earlier quoted context omitted.
It took (by my measure) 13 minutes for a full outage to be represented on the status page. I was under the impression that gitlab use gitlab.com for their work. Surely someone would have noticed within seconds that it was down? Why have the misleading "updated a few seconds" ago text if it doesn't update on complete failure? :)
After you notice I assume you have to declare an incident, get a call going, assess the extent of the issues, get the needed people involved, and then you'd announce on the status page. 13 minutes isn't amazing but it also isn't terrible. Perhaps you have better ways of keeping status pages updated much faster while also not ending up ramping up the posting of false positives.
13 minutes is pretty solid compared to anything of recent AWS outages.
Re: Gitlab is down
#38Earlier quoted context omitted.
Indeed, I can't remember a single time where a self-hosted server crashed. They run for decades with 0 downtime.
Well, my GitLab instance at some point started to have its Prometheus eat 100% CPU all the time until I disabled the Prometheus component altogether, so there’s that. A cursory glance at the tracker just now says the issue is still open. That’s the kind of problems you get for self-hosting, it’s not all rainbows and unicorns.
Hi, Developer Evangelist at GitLab here.
Can you link the issue please? :)
For context, Prometheus and observability will be handled with Opstrace in the future [0]. I'd like to learn about your use-case and see which troubles you have been running into. Thanks!
Re: Gitlab is down
#39Earlier quoted context omitted.
It took (by my measure) 13 minutes for a full outage to be represented on the status page. I was under the impression that gitlab use gitlab.com for their work. Surely someone would have noticed within seconds that it was down? Why have the misleading "updated a few seconds" ago text if it doesn't update on complete failure? :)
Your impression is correct. We use GitLab.com and notice these incidents as they happen. The delay in updating status is a result of our Incident Management process [0]. We have a Communications Manager on Call (CMOC) who leads communication throughout an incident. One of their responsibilities includes updating the status page. The slight delay between noticing the issue and updating the status page is a result of t…
Not a "status page" then, but merely "a page where Communications Manager post messages on after assessing the situation and consulting/getting permission from management"
Re: Gitlab is down
#40Earlier quoted context omitted.
Indeed, I can't remember a single time where a self-hosted server crashed. They run for decades with 0 downtime.
They could, if you stuck to the yak shaving full-time.
What? Running your own Gitlab instance is one docker command away. No need to shave any yaks.