Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gitlab was down

status.gitlab.com

Post reply on HN
Print view View original

41–50 of 140 posts

Re: Gitlab was down

#41

Post by Ndymium »

tadzik_ wrote: post #32
Uninen wrote: post #8

We use GitLab at work and I use it for personal projects as well, it has been very slow for several days now and they had downtime yesterday as well. I don't mind the general sluggishness of the system that much (as I love the platform in general) but when you can't get your work (nor hobbies) done because of tools breaking, it gets really annoying really fast.

> when you can't get your work (...) done because of tools breaking, it gets really annoying really fast I always assumed that the publicly hosted version of gitlab is basically a giant demo version of the enterprise edition you buy to host it yourself. Hell, you can even host the community edition for free. If your work relies on it, why rely on a free online product? EDIT: TIL gitlab.com also has a paid options. I…

gitlab.com is not only for free clients, plenty of paid clients use it instead of hosting themselves.

Re: Gitlab was down

#42

Post by ownagefool »

kburman wrote: post #4

> Website, API, Git (ssh and https), Pages, Registry, CI/CD, Background Processing, Support Services, packages.gitlab.com, customers.gitlab.com, version.gitlab.com, forum.gitlab.com How come all of them are down all at once.

Likely just means they have a Single Point of Failure.

Some guesses would be:

Automation/orchestration - They've migrated to k8s (I don't believe they've actually done this yet), but it could be their orchestration / automation tool automated a broken thing everywhere.

Database/Auth - Pretty much everything in gitlab will touch the database as far as I'm aware. Otherwise, how do you check whether users are auth'd to take action something. You wouldn't expect this to break the static website, i.e. the sales landing pages, but these could be based off an internal CMS, or could be checking for "guest" role session.

DNS/Service Discovery - As a sibling posted, "it's always DNS". It's good practice to use names for services instead of IP addresses, but this means your DNS needs to generally work, or everything will go down. Service Discovery could rely on DNS, but it could also be an API call that finds out DNS addresses or IP addresses directly.

CDN - You wouldn't typically put this in front of auth'd usage, and typically a CDN might not be helpful in front of something like SSH, but a quick look at fastly suggests they might support this. The main downside is sharing all the user data / auth tokens.

Security Product / CA - All you need is a requirement to encrypt internal traffic and rotate secrets, and you end up with a secret store that sits in the middle of everything.

Storage Layer - I believe they were big on Ceph for a while. If everything is backed by Ceph, everything will go down if you fail with Ceph.

Obviously, whatever it is, you'd expect them to split up their fail over plan a bit more in the future if it is something like that, but usually there's a single point of failure somewhere.

Re: Gitlab was down

#43

Post by softwarelimits »

dijit wrote: post #26

Notably they don’t hire any real people with ops experience. Erring instead to go for developers hoping that they can do everything needed. I like gitlab as a product but they don’t have a service mindset, and I think not hiring operations-centric people is a symptom of that which causes these kinds of issues.

How do you know that they do not hire "any people with ops experence"? Do you have insight? Is there any public evidence for this or anything you could publish yourself to add some substance to your words?

It would be great if you understand that just saying something is not enough on the internet.

Re: Gitlab was down

#44

Post by spzb »

the-dude wrote: post #25

It is worth mentioning their status page is not down.

It looks like it's externally hosted at status.io. It makes sense to have your status page on separate hosting otherwise it can go down at the same time as your main infrastructure and that defeats the point of having a status page.

Re: Gitlab was down

#45

Post by bamboozled »

dijit wrote: post #26

Notably they don’t hire any real people with ops experience. Erring instead to go for developers hoping that they can do everything needed. I like gitlab as a product but they don’t have a service mindset, and I think not hiring operations-centric people is a symptom of that which causes these kinds of issues.

A friend of mine interviewed for some kind of ops role there.

He was asked to clone a repo during the interview, he told me it was almost a GB.

When he commented it was slow compared to Github, the engineer interviewing him told him it was his fault for living in Australia and that they have bad internet and they're too far away from everything.

Go figure.

Re: Gitlab was down

#46

Post by greens231 »

Originally joined gitlab for their free private repos but with the recent downtime/sluggishness, i have jumped over to github (now that they offer free private repos too)

Re: Gitlab was down

#47

Post by andrelaszlo »

tadzik_ wrote: post #32
Uninen wrote: post #8

We use GitLab at work and I use it for personal projects as well, it has been very slow for several days now and they had downtime yesterday as well. I don't mind the general sluggishness of the system that much (as I love the platform in general) but when you can't get your work (nor hobbies) done because of tools breaking, it gets really annoying really fast.

> when you can't get your work (...) done because of tools breaking, it gets really annoying really fast I always assumed that the publicly hosted version of gitlab is basically a giant demo version of the enterprise edition you buy to host it yourself. Hell, you can even host the community edition for free. If your work relies on it, why rely on a free online product? EDIT: TIL gitlab.com also has a paid options. I…

> If your work relies on it, why rely on a free online product?

Some orgs are paying 99$ per user-month for the SaaS version of gitlab.com, they probably rely on it pretty heavily :)

https://web.archive.org/web/20191119174637/https://about.git...

Edit: others made the same point, but I'll leave this here for the link to the pricing page.

Re: Gitlab was down

#49

Post by Dayshine »

greens231 wrote: post #46

Originally joined gitlab for their free private repos but with the recent downtime/sluggishness, i have jumped over to github (now that they offer free private repos too)

Gitlab is still the only viable choice for non-commercial groups who want private repos though, the 3 private members and no ability to have mixtures of public/private repos in organisations on Github is very limiting.

Re: Gitlab was down

#50

Post by m712 »

kburman wrote: post #4

> Website, API, Git (ssh and https), Pages, Registry, CI/CD, Background Processing, Support Services, packages.gitlab.com, customers.gitlab.com, version.gitlab.com, forum.gitlab.com How come all of them are down all at once.

The status page has updated to indicate that they misconfigured their firewall. Apparently their entire set of services go through a single firewall (or at least, multiple firewalls with the same config). It's worrying that they don't have a staging setup for these kinds of things.

(NOTE: I am speculating here, if they do have a staging system and this wasn't reproduced there then the last sentence doesn't apply.)

Post reply on HN