Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gitlab.com was experiencing elevated error rates for Git, Web, and API

status.gitlab.com

Post reply on HN
Print view View original

81–90 of 94 posts

Re: Gitlab.com was experiencing elevated error rates for Git, Web, and API

#81

Post by hamburglar »

SkyPuncher wrote: post #72
gilfoyle wrote: post #51

Gitlab is a great service and a great alternative to github and bitbucket. But it has been quite flaky over many years and it's quite surprising they haven't prioritised system stability over everything else. https://twitter.com/gitlabstatus Gitlab has been a fully remote proponent going back a few years and these persistent reliability issues add some doubt to that model - even if there might be altogether other iss…

Fully remotes has nothing to do with it. My company is full remote and we don't have these issue.

Agreed. Blaming a remote work culture is a complete non sequitur.

Re: Gitlab.com was experiencing elevated error rates for Git, Web, and API

#82

Post by ahachete »

For those curious: the root cause for the incident is, ultimately, degraded statistics on a table.

There are a set of high frequency (very frequently run queries) that are quite sensible to plan flipping (they normally run with a given execution plan; if the plan changes to a worse one, effects can be dramatic, given their frequency).

That leads to a lot of query timeouts (GitLab's database limit query execution time to prevent further damage), which are visualized as database errors. This in turn leads to partial or complete downtime (effectively, if the database is running few effective --non timed out-- traffic).

Some of these queries depend on a particular Postgres planner way of working that is less than ideal (in PG11, it is improved in PG12), and may lead to plan flipping. Which in turn, happens when table statistics degrade. In GitLab's case, they are normally close to the threshold, and slight statistics degradation caused a plan change, which in turn lead to many queries time out and excessive load on the fleet.

At the end of the day, the fix is quite simple, however: update the table statistics (running ANALYZE). For more information: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3...

Re: Gitlab.com was experiencing elevated error rates for Git, Web, and API

#83

Post by dnsmichi »

sytse wrote: post #53
perihelions wrote: post #50

Earlier quoted context omitted.

Out of curiosity, why does your "about" page vouch for a suspended Twitter account? Is @GitLab -> @GitLabs a typo? https://about.gitlab.com/handbook/marketing/community-relati... @GitLab

Thanks! It is a typo and I submitted a fix https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request... Please note that about.gitlab.com contains all our static content, including our 10,000 page handbook. This typo isn’t on our main about page but deep in the marketing handbook.

Thanks Sid, merged :) We've also fixed the formatting at https://about.gitlab.com/handbook/marketing/community-relati... in https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_request...

Re: Gitlab.com was experiencing elevated error rates for Git, Web, and API

#84

Post by aaronharnly »

qeternity wrote: post #37

We switched from self-hosted to gitlab.com a couple months back and the migration has been extremely disappointing. Pushing commits often hangs completely, and we've had a number of smaller downtimes/degradations in recent weeks that have been super disruptive. Having been Gitlab users since our inception, we're now seriously considering Github (I know - grass is always greener).

What prompted you to move away from self-hosted? What was the maintenance overhead like?

Re: Gitlab.com was experiencing elevated error rates for Git, Web, and API

#85

Post by danaris »

Xelbair wrote: post #70
pbalau wrote: post #66

Earlier quoted context omitted.

We are using bitbucket atm and I can't fathom a reason to move, care to elaborate?

Atlassian is Australian company. Australia has very weird and wide reaching laws regarding software backdoors.

Can you elaborate on a) what kinds of laws on software backdoors? (Mandating them? Forbidding them? What sort?), and b) how that affects the GP's request?

Edit: Nvm on (b); somehow I got GitLab and BitBucket mixed up as to which one belonged to Atlassian.

Still curious about what sort of backdoor laws they have.

Re: Gitlab.com was experiencing elevated error rates for Git, Web, and API

#86

Post by danaris »

0x008 wrote: post #7
mondss wrote: post #2

I can't be the only one that feels that GitLab has extended downtime at least weekly, sometimes bi-weekly.

I use gitlab daily and have not experienced any outage in last 6 months probably.

I use GitLab for my side project (source control only; don't have any kind of CI set up just at the moment) and it's been years since there was any really noticeable slowness for me. I can't say I use it daily, but fairly frequently.

Re: Gitlab.com was experiencing elevated error rates for Git, Web, and API

#87

Post by herodoturtle »

danaris wrote: post #85
Xelbair wrote: post #70

Earlier quoted context omitted.

Atlassian is Australian company. Australia has very weird and wide reaching laws regarding software backdoors.

Can you elaborate on a) what kinds of laws on software backdoors? (Mandating them? Forbidding them? What sort?), and b) how that affects the GP's request? Edit: Nvm on (b); somehow I got GitLab and BitBucket mixed up as to which one belonged to Atlassian. Still curious about what sort of backdoor laws they have.

I think it's in reference to this from a few years ago: https://www.wired.com/story/australia-encryption-law-global-...

EDIT: I've done some more reading up on this, and The Assistance and Access Act of 2018 explicitly states that government cannot use backdoors: https://www.homeaffairs.gov.au/about-us/our-portfolios/natio...

Re: Gitlab.com was experiencing elevated error rates for Git, Web, and API

#88

Post by hobs »

ahachete wrote: post #82

For those curious: the root cause for the incident is, ultimately, degraded statistics on a table. There are a set of high frequency (very frequently run queries) that are quite sensible to plan flipping (they normally run with a given execution plan; if the plan changes to a worse one, effects can be dramatic, given their frequency). That leads to a lot of query timeouts (GitLab's database limit query execution time…

s/sensible/sensitive/

Would this be a place where a plan hint would help? It seems like having to re-analyze would mean that this is just a matter of time before it becomes problematic again.

Re: Gitlab.com was experiencing elevated error rates for Git, Web, and API

#89

Post by ahachete »

hobs wrote: post #88
ahachete wrote: post #82

For those curious: the root cause for the incident is, ultimately, degraded statistics on a table. There are a set of high frequency (very frequently run queries) that are quite sensible to plan flipping (they normally run with a given execution plan; if the plan changes to a worse one, effects can be dramatic, given their frequency). That leads to a lot of query timeouts (GitLab's database limit query execution time…

s/sensible/sensitive/ Would this be a place where a plan hint would help? It seems like having to re-analyze would mean that this is just a matter of time before it becomes problematic again.

> s/sensible/sensitive/

Thanks :)

> Would this be a place where a plan hint would help?

It is one option to consider (I wrote about it last week: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3...), but it is not an ideal solution, for several reasons.

Re-analyze is not the solution either, but as a short-term measure, cron-ed ANALYZEs will be run. Longer term, apart from refactoring some queries which are quite prone to trigger this plan behavior, statistics gathering process is going to be fully reviewed.

Re: Gitlab.com was experiencing elevated error rates for Git, Web, and API

#90

Post by kevinob11 »

INTPenis wrote: post #33

I love Gitlab because I love open source and competition but Gitlab either on-prem or gitlab.com has been having growing pains ever since I started using it around 2015. And it seems to be related to scaling because it's always iffy problems like higher error rates or jobs not executing. Either way, I love our on-prem Gitlab. It's free and it hosts over 200 projects, Gitops and the whole shebang. I'm just now beginni…

+1 for on-prem. I tried gitlab.com first but it was very slow (this was years ago, YMMV) but on-prem has been rock-solid and fast for us hosting ~100 projects. One thing I've been surprised by is how easy updates have been, I don't think we've ever had to even reconfigure anything, including runners. I know it seems like that is how it should work, but I've had a lot of opposite experiences with the open source tier of tools.
Post reply on HN