Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gitlab was down

status.gitlab.com

Post reply on HN
Print view View original

111–120 of 140 posts

Re: Gitlab was down

#111

Post by lbotos »

protomikron wrote: post #76
kungtotte wrote: post #73

Earlier quoted context omitted.

Whenever GitHub is down you see comments saying much the same thing only with the names reversed. The thing that strikes me each time is how fragile everyone's setup is if GitHub/lab is a single point of failure for them... At least gitlab let's you self host, which would let you run backups on offsite hosting meaning zero downtime.

GitHub's uptime is way better than GitLab. I am not really sure gitlab.com even reaches 99% availability (not to mention 99.9%). GitLab may well be focused on providing full-stack dev-services (VCS, build-server, CI and stuff), but in the end they are a hosting company - and for hosting, uptime is one of the most important metrics. EDIT: 99% was exaggerating, but I got so many 50Xs the last days that it was from time…

You can see GitLab.com's uptime here via pingdom:

http://stats.gitlab.com/4932705/2019/11 (99.75%)

http://stats.gitlab.com/4932705/2019/10 (99.98%)

http://stats.gitlab.com/4932705/2019/09 (100%)

http://stats.gitlab.com/4932705/2019/08 (100%)

http://stats.gitlab.com/4932705/2019/07 (100%)

http://stats.gitlab.com/4932705/2019/06 (99.98%)

EDIT: I see one of my colleagues also posted here, I wasn't asked to and I'm doing it of my own accord. I assume they are as well, but I can only speak for myself.

Re: Gitlab was down

#112

Post by appkate »

yRetsyM wrote: post #33

There has been a theme of instability with Gitlab.com over the last week or two. I'm not sure if it's growth related (they've seen a steady increase of users/traffic) and they've reached a scaling peak. OR if it's technically related - they've been doing a number of different infrastructure changes over the last few weeks which make a material difference to the main layers of the service. For me the real test here is…

Suffering the same instability issue and switched back to Github as that's what a paying customer would do.

Re: Gitlab was down

#113

Post by jeltz »

niceworkbuddy wrote: post #67
PhilKunz wrote: post #61

Earlier quoted context omitted.

I've always been a little cautious of running it myself. When I look at the components involved there are a lot of moving parts. And I don't want to be ever in a position where I can't get things started again. If I'm running on premise I'll switch to gitea with drone ci instead.

GitLab has virtual machine image, so it's really unobtrusive for your system.

They also have a Debian package with sets up the whole environment. It has worked well for me so far. The only thing I had to add was a backup script.

Re: Gitlab was down

#114

Post by Cthulhu_ »

kungtotte wrote: post #73
jlengrand wrote: post #66

I used to be a total Gitlab fanboy. I was going to Sid's meetups back in Utrecht 6 years ago, and he's a model of mine to this time. I would have done anything to stay on the platform, for at home and the office and setup gitlab instances in two different offices. Since somewhere last year, I moved back to Github primarily, and I'm sad to hear that my company is likely to make the same choice soon.The only reason is…

Whenever GitHub is down you see comments saying much the same thing only with the names reversed. The thing that strikes me each time is how fragile everyone's setup is if GitHub/lab is a single point of failure for them... At least gitlab let's you self host, which would let you run backups on offsite hosting meaning zero downtime.

Zero downtime is an ideal that you'd need a lot of investment for to achieve. I mean for 99.99999% uptime you'd already need to have a multi-server, multi-AZ, multi-region and multi-cloud setup and orchestrate data parity between those environments. That's not easy to set up and maintain.

Re: Gitlab was down

#115

Post by dijit »

toupeira wrote: post #101
dijit wrote: post #54

Earlier quoted context omitted.

What you say is completely fair. I’ve been looking at job postings and watching the way they work too for a little time since it’s all open. DBAs are “ruby devs who have used Postgres” https://about.gitlab.com/jobs/apply/backend-engineer-databas... SREs are “ruby devs who have used docker/kubernetes” (No job listing currently) The only open job labelled “ops” is telling. https://about.gitlab.com/jobs/apply/frontend-e…

GitLab engineer here. The "Ops" section actually consists of product development teams for features in GitLab itself (the Configure/Monitor "stages"), while the SREs are in the "Infrastructure" department. See https://about.gitlab.com/handbook/engineering/#engineering-d... . But yes, we don't seem to be hiring SREs at the moment, but I assume we'll add more openings in the beginning of next year. Regarding DBAs, that…

Hi Toupeira,

I didn't find some of those jobs, so thanks for linking them.

As a side note I just went through those job descriptions and I -really- like the layout.

On topic: Unfortunately they really do prove my point. There is a very strong focus on "strong" programming skills which is rather undefined. It's literally mentioned in every single role description.

The overwhelming majority of staff that knows how to run software reliably ironically are not software engineers, although there certainly are some software engineers who also possess this skill.

The people I'm speaking about typically understand concepts and solutions (like PAXOS, filesystems or public cloud) more than they understand software development methodology or software product structure.

I guess you have a global reach and can be quite picky about who you hire, maybe you /do/ exclusively hire architecture and systems focused programmers, or maybe "strong" programming skills are a different definition to mine.

Re: Gitlab was down

#116

Post by jeltz »

dijit wrote: post #85
tyingq wrote: post #55

Earlier quoted context omitted.

As recently as October 2017, they had exactly one database person. So I can imagine quite a lot of work and hiring needed to scale up a solid ops practice. "Until very recently I was the only database specialist which meant I had a lot of work on my plate." https://about.gitlab.com/blog/2017/10/02/scaling-the-gitlab-...

Almost to prove my point Yorick self-identifies as a "Ruby/Rust Developer" https://railsisrael2016.events.co.il/people/2644-yorick-pete... Obviously this doesn't preclude operations or systems knowledge but again, it's at least telling of the mindset.

To be fair it depends a bit on the context. I present myself sometimes as a database expert and other times a Ruby/C Developer. And I would argue that I, as a minor PostgreSQL contributor who follows the mailing lists, am more knowledgeable than most about databases and especially PostgreSQL.

Edit: Admittedly while I have done a lot of DBA stuff and server operations on top of my software development, hardware and networking are not my strong points so if the company could afford it I would want a more traditional server/networking guy on my team (and at a previous job I did exactly that). And I agree their job posting seem to have a very heavy focus on development experience.

Re: Gitlab was down

#117

Post by protomikron »

lbotos wrote: post #111
protomikron wrote: post #76

Earlier quoted context omitted.

GitHub's uptime is way better than GitLab. I am not really sure gitlab.com even reaches 99% availability (not to mention 99.9%). GitLab may well be focused on providing full-stack dev-services (VCS, build-server, CI and stuff), but in the end they are a hosting company - and for hosting, uptime is one of the most important metrics. EDIT: 99% was exaggerating, but I got so many 50Xs the last days that it was from time…

You can see GitLab.com's uptime here via pingdom: http://stats.gitlab.com/4932705/2019/11 (99.75%) http://stats.gitlab.com/4932705/2019/10 (99.98%) http://stats.gitlab.com/4932705/2019/09 (100%) http://stats.gitlab.com/4932705/2019/08 (100%) http://stats.gitlab.com/4932705/2019/07 (100%) http://stats.gitlab.com/4932705/2019/06 (99.98%) EDIT: I see one of my colleagues also posted here, I wasn't asked to and I'm doing…

Ok, I admit that 99% was exaggerating, but you sure should make availability of gitlab.com your main focus - and you promise that for years to the users (linking at issues etc.).

Btw.: Your links are http, and the https version has `SSL_ERROR_BAD_CERT_DOMAIN` - you should check with pingdom.com maybe?

Re: Gitlab was down

#118

Post by YorickPeterse »

dijit wrote: post #85
tyingq wrote: post #55

Earlier quoted context omitted.

As recently as October 2017, they had exactly one database person. So I can imagine quite a lot of work and hiring needed to scale up a solid ops practice. "Until very recently I was the only database specialist which meant I had a lot of work on my plate." https://about.gitlab.com/blog/2017/10/02/scaling-the-gitlab-...

Almost to prove my point Yorick self-identifies as a "Ruby/Rust Developer" https://railsisrael2016.events.co.il/people/2644-yorick-pete... Obviously this doesn't preclude operations or systems knowledge but again, it's at least telling of the mindset.

I would not call myself an operations expert by any means, but I have done quite a bit of infrastructure work in the past; both with bare metal setups and cloud based solutions.

With that said, the old database specialist position was about 80% engineering, 20% infrastructure, with the infrastructure work being done in cooperation with the production engineers.

I left the database team a good year ago and quite a lot has changed since then. I think these days we have a handful of people focusing on the database side of things.

Re: Gitlab was down

#119

Post by erikbye »

oefrha wrote: post #64
1337shadow wrote: post #51

Back in the days GitLab made it clear that it aspires to be a software company rather than an infrastructure company. The reason I favor GitLab over GitHub is that I can install it on a dedicated server and have blazing fast performance, which helps development performance on an every day basis. I've been maintaining a bunch of deployments of all sorts (k8s, docker-compose, baremetal) for the last years for several c…

> have blazing fast performance Sorry, but GitLab has never been blazing fast. The gitlab.com instance is notoriously slow — supposedly improved a lot over the past few years, but still feels pretty sluggish. My self-hosted instance isn’t much better. In fact, a couple of open source maintainer friends looked into migrating to GitLab when GitHub was acquired by MS; they did not precisely because GitLab was too slow.

Not to mention consuming way too much memory. I prefer Gitea, or cgit if I need even less features.

Re: Gitlab was down

#120

Post by kungtotte »

Cthulhu_ wrote: post #114
kungtotte wrote: post #73

Earlier quoted context omitted.

Whenever GitHub is down you see comments saying much the same thing only with the names reversed. The thing that strikes me each time is how fragile everyone's setup is if GitHub/lab is a single point of failure for them... At least gitlab let's you self host, which would let you run backups on offsite hosting meaning zero downtime.

Zero downtime is an ideal that you'd need a lot of investment for to achieve. I mean for 99.99999% uptime you'd already need to have a multi-server, multi-AZ, multi-region and multi-cloud setup and orchestrate data parity between those environments. That's not easy to set up and maintain.

Yes.

Tell that to all the companies that have to stop working when either of these sites go down because they've built their entire workflow on GitHub/lab being up.

Since zero downtime is so costly, maybe it's a better idea not to rely on the uptime of a third party?

Post reply on HN