Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gitlab was down

status.gitlab.com

Post reply on HN
Print view View original

51–60 of 140 posts

Re: Gitlab was down

#51

Post by 1337shadow »

Back in the days GitLab made it clear that it aspires to be a software company rather than an infrastructure company.

The reason I favor GitLab over GitHub is that I can install it on a dedicated server and have blazing fast performance, which helps development performance on an every day basis.

I've been maintaining a bunch of deployments of all sorts (k8s, docker-compose, baremetal) for the last years for several customers of all sorts of size, the upgrade process has always been pretty smooth, sometimes I ran in edgecases yes but always found a solution.

My deployments are always up and kicking, unless I'm messing with the configuration and doing some mistakes.

I highly recommend hosting your own GitLab instance, even on a single server.

Re: Gitlab was down

#52

Post by toopok4k3 »

PhilKunz wrote: post #18

It seems like GitLab should indeed focus on stability now. The feature set is great. But it is all not worth it if it keeps disrupting work on a constant basis by being unstable.

I recommend hosting it yourself, works great and is fast on local network. You can run updates when it's convenient to you, not to others.

GitLab is a rare software that enables you to be in control.

Re: Gitlab was down

#53

Post by 11235813213455 »

Uninen wrote: post #8

We use GitLab at work and I use it for personal projects as well, it has been very slow for several days now and they had downtime yesterday as well. I don't mind the general sluggishness of the system that much (as I love the platform in general) but when you can't get your work (nor hobbies) done because of tools breaking, it gets really annoying really fast.

In terms of energy and storage usage (so environment footprint), wouldn't their shared online hosting be more efficient than many self-hosted ones here and there?

Re: Gitlab was down

#54

Post by dijit »

softwarelimits wrote: post #43
dijit wrote: post #26

Notably they don’t hire any real people with ops experience. Erring instead to go for developers hoping that they can do everything needed. I like gitlab as a product but they don’t have a service mindset, and I think not hiring operations-centric people is a symptom of that which causes these kinds of issues.

How do you know that they do not hire "any people with ops experence"? Do you have insight? Is there any public evidence for this or anything you could publish yourself to add some substance to your words? It would be great if you understand that just saying something is not enough on the internet.

What you say is completely fair.

I’ve been looking at job postings and watching the way they work too for a little time since it’s all open.

DBAs are “ruby devs who have used Postgres”

https://about.gitlab.com/jobs/apply/backend-engineer-databas...

SREs are “ruby devs who have used docker/kubernetes”

(No job listing currently)

The only open job labelled “ops” is telling.

https://about.gitlab.com/jobs/apply/frontend-engineer---conf...

Re: Gitlab was down

#55

Post by tyingq »

softwarelimits wrote: post #43
dijit wrote: post #26

Notably they don’t hire any real people with ops experience. Erring instead to go for developers hoping that they can do everything needed. I like gitlab as a product but they don’t have a service mindset, and I think not hiring operations-centric people is a symptom of that which causes these kinds of issues.

How do you know that they do not hire "any people with ops experence"? Do you have insight? Is there any public evidence for this or anything you could publish yourself to add some substance to your words? It would be great if you understand that just saying something is not enough on the internet.

As recently as October 2017, they had exactly one database person. So I can imagine quite a lot of work and hiring needed to scale up a solid ops practice.

"Until very recently I was the only database specialist which meant I had a lot of work on my plate."

https://about.gitlab.com/blog/2017/10/02/scaling-the-gitlab-...

Re: Gitlab was down

#56

Post by ownagefool »

ownagefool wrote: post #42
kburman wrote: post #4

> Website, API, Git (ssh and https), Pages, Registry, CI/CD, Background Processing, Support Services, packages.gitlab.com, customers.gitlab.com, version.gitlab.com, forum.gitlab.com How come all of them are down all at once.

Likely just means they have a Single Point of Failure. Some guesses would be: Automation/orchestration - They've migrated to k8s (I don't believe they've actually done this yet), but it could be their orchestration / automation tool automated a broken thing everywhere. Database/Auth - Pretty much everything in gitlab will touch the database as far as I'm aware. Otherwise, how do you check whether users are auth'd to…

Replying to myself, because it's now on their status page that a firewall change took down the database.

This points to there being:

- a lack of process and testing on key networking changes. Aren't they doing CI/CD, automated testing and peer review for this?

- A SPOF in the database; why couldn't things connect to a secondary for a read-only mode?

Quite a lot of the time, things break for stupid reasons. The main difference is when a normal company does something stupid, they can hide it, lie about it, or make it sound more complex.

The fact Gitlab publishes their fuck ups, is supposed to force them to do a better job and actually look at root causes and apply proper fixes that we can all judge. I wouldn't hold any particular fuck-up against them.

Re: Gitlab was down

#57

Post by dordoka »

beaconstudios wrote: post #7
kburman wrote: post #4

> Website, API, Git (ssh and https), Pages, Registry, CI/CD, Background Processing, Support Services, packages.gitlab.com, customers.gitlab.com, version.gitlab.com, forum.gitlab.com How come all of them are down all at once.

presumably a single point of failure - my guess would be something at the network level.

Spot on. The status page confirms it's a bad firewall configuration

Re: Gitlab was down

#58

Post by wbl »

ownagefool wrote: post #56
ownagefool wrote: post #42

Earlier quoted context omitted.

Likely just means they have a Single Point of Failure. Some guesses would be: Automation/orchestration - They've migrated to k8s (I don't believe they've actually done this yet), but it could be their orchestration / automation tool automated a broken thing everywhere. Database/Auth - Pretty much everything in gitlab will touch the database as far as I'm aware. Otherwise, how do you check whether users are auth'd to…

Replying to myself, because it's now on their status page that a firewall change took down the database. This points to there being: - a lack of process and testing on key networking changes. Aren't they doing CI/CD, automated testing and peer review for this? - A SPOF in the database; why couldn't things connect to a secondary for a read-only mode? Quite a lot of the time, things break for stupid reasons. The main d…

Network devices are generally hostile to advanced automation, and if they had both primary and secondary as the same class of machine then the changes would apply to both.

Re: Gitlab was down

#60

Post by Uninen »

tadzik_ wrote: post #32
Uninen wrote: post #8

We use GitLab at work and I use it for personal projects as well, it has been very slow for several days now and they had downtime yesterday as well. I don't mind the general sluggishness of the system that much (as I love the platform in general) but when you can't get your work (nor hobbies) done because of tools breaking, it gets really annoying really fast.

> when you can't get your work (...) done because of tools breaking, it gets really annoying really fast I always assumed that the publicly hosted version of gitlab is basically a giant demo version of the enterprise edition you buy to host it yourself. Hell, you can even host the community edition for free. If your work relies on it, why rely on a free online product? EDIT: TIL gitlab.com also has a paid options. I…

> If your work relies on it, why rely on a free online product?

We're not using the free product, we're paying customers (using the hosted saas version of the product).

Post reply on HN