Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gitlab is down

gitlab.com

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Gitlab is down

#21

Post by shakna »

ploggingdev wrote: post #20

I get the impression that Gitlab focuses a little too much on releasing new features at a rapid pace with every release. Maybe they should spend more time on running their infrastructure reliably and getting their engineering practices up to speed. The recent events will raise a red flag with enterprises who might be potential Gitlab customers and that directly affects their bottomline.

Gitlab.com is for testing new features. It's a testing environment.

If anything, enterprises can rest easy, because the next version has already been running in the wild, with a real world group of users running on it.

This was a first RC. It's fairly normal that infrastructure problems can arise here.

Re: Gitlab is down

#22

Post by jschulenklopper »

mdekkers wrote: post #14
foobazzy wrote: post #4

status.gitlab.com is also not loading. Gitlab pages hasn't been working properly for me(redirects to a 404)[1] [1]: https://gitlab.com/gitlab-org/gitlab-pages/issues/43

running your status page from the same infrastructure is a really bad idea

It is, in general. But I've seen no indication that such is the case for Gitlab.

Au contraire, status.gitlab.com seems to be located in NYC (in a Digital Ocean DC), and www.gitlab.com somewhere in Virginia (at Microsoft Azure?).

Re: Gitlab is down

#23

Post by wears_sweaters1 »

shakna wrote: post #17
jameskegel wrote: post #11

I'm rapidly starting to question my use of mid-tier web services. Who else is operating like this? CI/CD, Staging, downtime playbooks, backup playbooks, all of this or any combination of it would have been a good idea. Folks, I just want to work without my tools failing so that I can go home and think about something else.

GitLab.com is a testing platform for their Enterprise version. If you want to guarantee reliability you need to pay for hosting or self-host. Otherwise, there are quite a few competitors in this market with 99.99%+ guarantees.

Does the enterprise product move slower?

Re: Gitlab is down

#24

Post by scaryclam »

shakna wrote: post #17
jameskegel wrote: post #11

I'm rapidly starting to question my use of mid-tier web services. Who else is operating like this? CI/CD, Staging, downtime playbooks, backup playbooks, all of this or any combination of it would have been a good idea. Folks, I just want to work without my tools failing so that I can go home and think about something else.

GitLab.com is a testing platform for their Enterprise version. If you want to guarantee reliability you need to pay for hosting or self-host. Otherwise, there are quite a few competitors in this market with 99.99%+ guarantees.

If this is even remotely true, they need to put that on the front page, in really big letters.

Re: Gitlab is down

#25

Post by revelation »

ploggingdev wrote: post #20

I get the impression that Gitlab focuses a little too much on releasing new features at a rapid pace with every release. Maybe they should spend more time on running their infrastructure reliably and getting their engineering practices up to speed. The recent events will raise a red flag with enterprises who might be potential Gitlab customers and that directly affects their bottomline.

The real gem (pun intended) of Gitlab is not Gitlab.com, but the open source self-hosted version.

Chances are I don't want them to improve their infrastructure to the point where it can be a Github 2.0, because that probably means the setup and ongoing maintenance requirements of the self-hosted version will become excessive to support a scale that nobody using it has.

Re: Gitlab is down

#26

Post by overcast »

Gitlab is praised for being transparent in everything they do, so where is the backup infrastructure policy that they should now have in place? I'd like to see that situation proven resolved before we discuss rewriting their front end with Vue.js and any other new deployments.

Re: Gitlab is down

#27

Post by tvmalsv »

mdekkers wrote: post #14
foobazzy wrote: post #4

status.gitlab.com is also not loading. Gitlab pages hasn't been working properly for me(redirects to a 404)[1] [1]: https://gitlab.com/gitlab-org/gitlab-pages/issues/43

running your status page from the same infrastructure is a really bad idea

Doesn't look like they are, to me. Their main domain's IP address is owned by Microsoft (so, using Azure?), but the status page IP is in a block owned by Digital Ocean.

Edit: I could have sworn I refreshed the page before replying to make sure someone else hadn't already responded, and I didn't see your comment jschulenklopper. Scary how similar they are lol.

Re: Gitlab is down

#28

Post by swozey »

shakna wrote: post #21
ploggingdev wrote: post #20

I get the impression that Gitlab focuses a little too much on releasing new features at a rapid pace with every release. Maybe they should spend more time on running their infrastructure reliably and getting their engineering practices up to speed. The recent events will raise a red flag with enterprises who might be potential Gitlab customers and that directly affects their bottomline.

Gitlab.com is for testing new features. It's a testing environment. If anything, enterprises can rest easy, because the next version has already been running in the wild, with a real world group of users running on it. This was a first RC. It's fairly normal that infrastructure problems can arise here.

Yeah, I run Gitlab-EE internally and am aware of gitlab.com being a test bed but I can definitely see how others less familiar with Gitlab aren't aware of that.

Taking a quick glance at their site I can't even find anything about it basically being the test bed. There used to be some lines last year that said something along the lines that gitlab.com was known to be unstable and they recommended important projects to be self hosted. Maybe it was in the docs, I forget.

They should probably emphasize that somewhere if that's still the route they're going, although I feel like I did read they're working on major stability/infrastructure upgrades to solve those issues.

I don't pay much attention to gitlab.com since I use my own ee.

Re: Gitlab is down

#29

Post by justinlaster »

michaelt wrote: post #18
amingilani wrote: post #8

Question: with a strict CI/CD in place, as well as a staging server, how can these problems be so common for Gitlab? Isn't this exactly what CI is supposed to prevent? Not blaming Gitlab for bad practices or anything, i'm just curious.

Even with a staging server, things can pass testing but fail in production if the staging environment provides an imperfect simulation of the production environment - and that's almost inevitable. For example, your staging environment servers should be connecting to a different database with a different password. If the password's right in the staging config but wrong (or missing) in the production config, things tha…

> things can pass testing but fail in production if the staging environment provides an imperfect simulation of the production environment

Your staging environment should match production, or it's not really staging at that point. It doesn't have to match it in _size_, just structure and process. Ignoring data loss, if you can't quickly switch staging to production it's not really staging. It's just a dorky test environment masquerading as a stage environment. It's also surprisingly not that difficult (the variation of difficulty depends on the type of data you're interacting with, and how isolated it needs to be) to "forward" a slice of real word traffic to your staging environment and monitor it for some duration of time.

>For example, your staging environment servers should be connecting to a different database with a different password.

Handled by proper CI/CD pipelines. Completely irrelevant to deploying new features, configuration for production specific users/passwords happens on the sysadmin/devops side of things.

Re: Gitlab is down

#30

Post by swozey »

wears_sweaters1 wrote: post #23
shakna wrote: post #17

Earlier quoted context omitted.

GitLab.com is a testing platform for their Enterprise version. If you want to guarantee reliability you need to pay for hosting or self-host. Otherwise, there are quite a few competitors in this market with 99.99%+ guarantees.

Does the enterprise product move slower?

If you mean release-wise it's in step with CE. I host an ee server. I've had very few issues with the actual hosting of it over the last year. Their Omnibus system is fantastic. Most issues are UX/UI related when they break a button or what not.
Post reply on HN