Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gitlab is down

gitlab.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: Gitlab is down

#51

Post by oefrha »

dnsmichi wrote: post #38
oefrha wrote: post #32

Earlier quoted context omitted.

Well, my GitLab instance at some point started to have its Prometheus eat 100% CPU all the time until I disabled the Prometheus component altogether, so there’s that. A cursory glance at the tracker just now says the issue is still open. That’s the kind of problems you get for self-hosting, it’s not all rainbows and unicorns.

Hi, Developer Evangelist at GitLab here. Can you link the issue please? :) For context, Prometheus and observability will be handled with Opstrace in the future [0]. I'd like to learn about your use-case and see which troubles you have been running into. Thanks! [0] https://opstrace.com/blog/gitlab

This one I guess? https://gitlab.com/gitlab-org/gitlab/-/issues/210365

To be clear, I don't care about Prometheus on my instance, I only care when it's causing trouble (e.g. by eating 100% CPU all the time).

Re: Gitlab is down

#52

Post by john_cogs »

linuxdeveloper wrote: post #45
john_cogs wrote: post #33

Earlier quoted context omitted.

Your impression is correct. We use GitLab.com and notice these incidents as they happen. The delay in updating status is a result of our Incident Management process [0]. We have a Communications Manager on Call (CMOC) who leads communication throughout an incident. One of their responsibilities includes updating the status page. The slight delay between noticing the issue and updating the status page is a result of t…

Why is "Active Incident" and "System Wide Outage" on the status page with a background color of green? Why not red? At first glance it looks like everything is operational with no issues.

The color is green because GitLab.com is accessible again.

"Active Incident" remains because our team is still working towards full recovery.

"System Wide Outage" is the description of the incident at its onset.

Re: Gitlab is down

#53

Post by analogsalad »

manquer wrote: post #46
analogsalad wrote: post #14

Earlier quoted context omitted.

Indeed, I can't remember a single time where a self-hosted server crashed. They run for decades with 0 downtime.

It doesn't, but I can fix it as opposed to waiting for their team to do it. Also at gitlab.com scale the problems they face are very different from a typical deployment. It is like having maintaining your car and using the train. On average if you can fix your car (or hire a good mechanic i.e. consulting) you would probably have a better experience than public transport breaking down, that you are powerless to do any…

As a customer of Gitlab, I'm satisfied with their uptime and I have no reason to believe that they can't fix these issues in good time.

Yes, I can also fix it if the server was my mine but more than likely I'll be busy doing my actual job (which does not involve fiddling with self-hosted gitlab instances) so I'll take my chances with the Gitlab engineering team. They do fix things and me being busy, asleep, sick, or travelling have no impact on their response. I intend to keep it this way.

Re: Gitlab is down

#54

Post by miltonlaxer »

rvz wrote: post #23
analogsalad wrote: post #14

Earlier quoted context omitted.

Indeed, I can't remember a single time where a self-hosted server crashed. They run for decades with 0 downtime.

Exactly. That is the whole point. I keep telling that for GitHub since that goes down once a month. [0][1] GitLab SaSS is the same but a self-hosted backup is better. [0] https://news.ycombinator.com/item?id=29901564 [1] https://news.ycombinator.com/item?id=29379648

FYI GitHub Enterprise can be self-hosted https://docs.github.com/en/get-started/learning-about-github...

Re: Gitlab is down

#55

Post by manquer »

analogsalad wrote: post #53
manquer wrote: post #46

Earlier quoted context omitted.

It doesn't, but I can fix it as opposed to waiting for their team to do it. Also at gitlab.com scale the problems they face are very different from a typical deployment. It is like having maintaining your car and using the train. On average if you can fix your car (or hire a good mechanic i.e. consulting) you would probably have a better experience than public transport breaking down, that you are powerless to do any…

As a customer of Gitlab, I'm satisfied with their uptime and I have no reason to believe that they can't fix these issues in good time. Yes, I can also fix it if the server was my mine but more than likely I'll be busy doing my actual job (which does not involve fiddling with self-hosted gitlab instances) so I'll take my chances with the Gitlab engineering team. They do fix things and me being busy, asleep, sick, or…

Nothing wrong with that outlook, plenty of people do take the train after all extending the analogy further.

However ridiculing people who want to their control infrastructure better because they don't have the expertise or time as the guys running the railway gets old.

Re: Gitlab is down

#56

Post by analogsalad »

manquer wrote: post #55
analogsalad wrote: post #53

Earlier quoted context omitted.

As a customer of Gitlab, I'm satisfied with their uptime and I have no reason to believe that they can't fix these issues in good time. Yes, I can also fix it if the server was my mine but more than likely I'll be busy doing my actual job (which does not involve fiddling with self-hosted gitlab instances) so I'll take my chances with the Gitlab engineering team. They do fix things and me being busy, asleep, sick, or…

Nothing wrong with that outlook, plenty of people do take the train after all extending the analogy further. However ridiculing people who want to their control infrastructure better because they don't have the expertise or time as the guys running the railway gets old.

I had no intention of ridiculing anybody, there are things that I self-host but I like to pick and choose my battles instead of a blanket "I must have control over everything" approach. My response was specifically to this comment:

> And this is why you self-host on your own instance.

This was the commentary on the outage, and it's just outright wrong. Your self-hosted instances will also experience outages. That's the point I inteded to raise.

Re: Gitlab is down

#57

Post by manquer »

analogsalad wrote: post #56
manquer wrote: post #55

Earlier quoted context omitted.

Nothing wrong with that outlook, plenty of people do take the train after all extending the analogy further. However ridiculing people who want to their control infrastructure better because they don't have the expertise or time as the guys running the railway gets old.

I had no intention of ridiculing anybody, there are things that I self-host but I like to pick and choose my battles instead of a blanket "I must have control over everything" approach. My response was specifically to this comment: > And this is why you self-host on your own instance. This was the commentary on the outage, and it's just outright wrong. Your self-hosted instances will also experience outages. That's t…

I meant the OP , who was making fun of self hosting, didn't mean to imply you did.

I am not denying that there different people have different needs, not everyone wants or has the time to drive either . Just that self hosting is also valid even with uptime or other concerns.

Having said that achieving better uptime is quite possible self hosting not because we are better but because we have simpler challenges than gitlab.com, poorer uptime is also quite likely if we don't know what we are doing.

Simply put, managed services doesn't mean better uptime doesn't automatically .

Re: Gitlab is down

#58

Post by nix23 »

analogsalad wrote: post #29
sdoering wrote: post #26

Earlier quoted context omitted.

Not sure if this is irony (I often don't identify irony as such). But I fatfingered a lot of self hosted stuff in my time.

It's full-blown sarcasm. Sorry for omitting the /s.

And i was thinking AHHH he has a old IBM mainframe at home ;)

Re: Gitlab is down

#60

Post by hackerlytest »

john_cogs wrote: post #7

GitLab team member here. We're aware of the incident and the status page has been updated. We will provide further updates on the status page as they become available. (Edited now that the status page has been updated).

Thanks. Seems to be back for me.
Post reply on HN