Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Loss at GitLab

blog.2ndquadrant.com

Post reply on HN
Print view View original

141–150 of 236 posts

Re: Data Loss at GitLab

#141

Post by JohannesH »

synicalx wrote: post #104

Honestly, I'm completely flabbergasted by this. Five backups, and NONE worked properly? Who made this? The S3 bucket was EMPTY? Has no one ever tested any of these backups? It's not just the impact, which is fairly sizeable in it's own right, but it's the HUGE oversight on their part and the fact they tried to pin part of this on PostreSQL? Credit where it's due; their report/transparency were good if a little unprof…

Untested backup === No backup

People like to say they have backups or a "backup procedure", but in my experience almost none of them ever tested the backup... Not even once. 95% of the time "having a backup procedure" just means "we have a replica of some data sitting somewhere with no idea how/if we can restore it, or how long it takes".

Re: Data Loss at GitLab

#142

Post by fictioncircle »

praneshp wrote: post #93
aamederen wrote: post #29

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

HN can be funny sometimes. GitHub got a lot of hate about a year ago just for not releasing new features. GitLab cost everyone a day because their backup/ops practices were silly, and everyone loves them more. I've screwed up before, and I sympathize/empathize with their ops folks, but this should make us think about plan B in case something like this happens again.

The "Plan B" is to have a local Gitlab instance and use the mirror feature to mirror it to Gitlab.com and/or vice versa.

You should always have 2+ production nodes in case one goes down.

Re: Data Loss at GitLab

#143

Post by JohannesH »

iatanasov wrote: post #116
nicpottier wrote: post #58

Earlier quoted context omitted.

I think you are confused. The best thing to happen to OSS is GitHub, not GitLab. GitLab is just a fast follower and likely wouldn't even exist without the former. I for one am happy to throw money GitHub's way for their role in so dramatically changing how we code.

I think you might be confused. One of the nice things to happen to OSS was the availability of kernel ( Linux kernel that is ) and git ( ... ) and both set very open models which allowed the OSS to trive. Github allowed for using git without knowing it .

I think you might be confused. One of the nice things to happen to OSS was the availability of transistors and electricity. Both made it possible to make millions of logic calculations per second without knowing it. :)

Re: Data Loss at GitLab

#144

Post by taneq »

BHSPitMonkey wrote: post #67
balls187 wrote: post #59

Earlier quoted context omitted.

Why? Everyone has their own copy of the gitlab remote.

Not everyone has their own copy of the project's issues/pull requests/settings/webhooks/CI config/etc., which the incident affected (and 6 hours worth were lost). Git repositories themselves do have the happy side effect that at least someone on your team has a full latest clone on their machine, but that's only because the source code is what you're there to use every day. Even if GitLab had a great "checkout your i…

Maybe a dumb question but couldn't they be included in a hidden folder in the root of your repo or something? That way they'd get replicated automatically every time someone does a pull.

Re: Data Loss at GitLab

#145

Post by baq »

JohannesH wrote: post #141
synicalx wrote: post #104

Honestly, I'm completely flabbergasted by this. Five backups, and NONE worked properly? Who made this? The S3 bucket was EMPTY? Has no one ever tested any of these backups? It's not just the impact, which is fairly sizeable in it's own right, but it's the HUGE oversight on their part and the fact they tried to pin part of this on PostreSQL? Credit where it's due; their report/transparency were good if a little unprof…

Untested backup === No backup People like to say they have backups or a "backup procedure", but in my experience almost none of them ever tested the backup... Not even once. 95% of the time "having a backup procedure" just means "we have a replica of some data sitting somewhere with no idea how/if we can restore it, or how long it takes".

it's worse. you might be trying to restore from an untested backup and fail, wasting precious time.

five times.

Re: Data Loss at GitLab

#146

Post by baq »

kakwa_ wrote: post #74
activatedgeek wrote: post #22

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

Well, even if their transparency in failure is admirable, I'm not completely surprised. Just running an apt-get install gitlab gives me around ~350 dependencies. In those dependencies, I see python, ruby, nodejs, redis, postgres. With a little Java, and a little Go, plus some admin scripts written in Perl, the picture would be mostly complete... I may be a little harsh, but when I see a piece of software with so much…

to be fair, nodejs is useful for npm even if you're only doing client-side js.

Re: Data Loss at GitLab

#147

Post by sdesol »

problems wrote: post #130
thewhitetulip wrote: post #111

Earlier quoted context omitted.

>It's popular because you can install your own instance for free practically anywhere with minimal effort. If that is the reason why you use Gitlab, then why not try gitea or gogs? gogs is written in Go and provides a docker image or a drop in binary

GitLab is a lot more feature packed than Gitea/Gogs. Gogs is lightweight, good for personal projects, but if you're looking for something to deploy company wide with integrated everything, Gitlab is the way to go.

I think this is changing at a pretty fast pace ... well for Gitea anyways. It also looks like Gitea is lighting a fire under Gogs, as they appear to be iterating at a faster pace as well. Here's a very quick breakdown of what's going on.

Activity for the last 160 days. There were 175 commits to gogs and 720 commits to gitea.

https://gitsense.com/gogs-gitea/commits-160days.png

Activity for the last 60 days. There were 109 commits to gogs and 262 to gitea.

https://gitsense.com/gogs-gitea/commits-60days.png

https://gitsense.com/gogs-gitea/changes-60days.png

https://gitsense.com/gogs-gitea/changes-files-60days.png

The options and vendors directory are unique to Gitea and they account for a lot of the changes within the last 60 days. I was told the vendors directory is used to store dependencies but I don't know what the options directory is used for. And as the following shows, they account for a lot of the files touched, in the last 60 days.

https://gitsense.com/gogs-gitea/changes-options-vendor-60day...

Based on what I've read on Hacker News, the developer behind Gogs, tends to merge in changes in spurts, so it's hard to tell if this recent flurry of activity is a spurt or not. In this 365 days of activity, you can see the 3 spurts for Gogs so far.

https://gitsense.com/gogs-gitea/commits-365days.png

Regardless of whether or not Gogs will continue to develop at an increased rated, it looks like Gitea will.

Re: Data Loss at GitLab

#148

Post by esseti »

Aside from the incident, this is a great opportunity to learn something new (at least for me). Said so, does anyone of you know how they plot all the charts of postgres and co? such as this: http://monitor.gitlab.net/dashboard/db/postgres-queries I know that the chart is made in grafana, but: how do they collect the data?

Re: Data Loss at GitLab

#149

Post by mdekkers »

beefsack wrote: post #86
balls187 wrote: post #59

Earlier quoted context omitted.

Why? Everyone has their own copy of the gitlab remote.

We use gitlab.com for code reviews and CI, and we basically lost a day of integration productivity because of it. Yes, our process is too tightly tied to a single service, but it happens because we don't have the resources to self-host our own solution. We love GitLab, but this has absolutely got us looking elsewhere.

The gitlab service is obviously important to your organisation, but not important enough to pay for their service, and incidentally, support a more robust environment?

Re: Data Loss at GitLab

#150

Post by mdekkers »

synicalx wrote: post #104

Honestly, I'm completely flabbergasted by this. Five backups, and NONE worked properly? Who made this? The S3 bucket was EMPTY? Has no one ever tested any of these backups? It's not just the impact, which is fairly sizeable in it's own right, but it's the HUGE oversight on their part and the fact they tried to pin part of this on PostreSQL? Credit where it's due; their report/transparency were good if a little unprof…

Putting on my BOFH hat, this is what happens when you let Devs do operational stuff.

:thumbsup:

Post reply on HN