Live data from Hacker News

Data Loss at GitLab

blog.2ndquadrant.com

161–170 of 236 posts

Re: Data Loss at GitLab

#161
post #148

Aside from the incident, this is a great opportunity to learn something new (at least for me). Said so, does anyone of you know how they plot all the charts of postgres and co? such as this: http://monitor.gitlab.net/dashboard/db/postgres-queries I know that the chart is made in grafana, but: how do they collect the data?

We use Prometheus prometheus.io We have a team of Prometheus engineers and have a vacancy for more.

Re: Data Loss at GitLab

#162
post #24
post #6

I don't get to work on databases this size and today has been an incredible lesson and a journey. I've been reading all the comments and blogs, watching the stream and Googling what I didn't know or understand. I feel like the next step for me is scaling my business so that we have an actual usage for my newly found interests :)

For those wondering about the live Gitlab stream, you can watch them work here: https://www.youtube.com/watch?v=nc0hPGerSd4

That is remarkably transparent. Commendable.

Re: Data Loss at GitLab

#163

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

HN can be funny sometimes. GitHub got a lot of hate about a year ago just for not releasing new features. GitLab cost everyone a day because their backup/ops practices were silly, and everyone loves them more. I've screwed up before, and I sympathize/empathize with their ops folks, but this should make us think about plan B in case something like this happens again.

Every corporation ever has screwed up like GitLab if not more.

They are just better at hiding it, smoothing it over and lying.

Re: Data Loss at GitLab

#164
post #83

Earlier quoted context omitted.

I think you are confused. The best thing to happen to OSS is GitHub, not GitLab. GitLab is just a fast follower and likely wouldn't even exist without the former. I for one am happy to throw money GitHub's way for their role in so dramatically changing how we code.

Github itself is not open source. The irony.

Yeah, even their enterprise edition code is obfuscated like crazy.

For a company that praises open-source so much they really do try to be as closed as possible themselves.

Re: Data Loss at GitLab

#165

Earlier quoted context omitted.

Why? Everyone has their own copy of the gitlab remote.

We use gitlab.com for code reviews and CI, and we basically lost a day of integration productivity because of it. Yes, our process is too tightly tied to a single service, but it happens because we don't have the resources to self-host our own solution. We love GitLab, but this has absolutely got us looking elsewhere.

Do you self-host any services currently & have they incurred any similar unexpected downtime in the past?

If you self-hosted code reviews/CI, would you expect having a similar downtime causing problem in the future?

I expect the answer to be "Yes" for most companies.

Re: Data Loss at GitLab

#166

Earlier quoted context omitted.

There not many project these days that would not require similar stacks. At least one core language: be it Ruby, or Java or whatever. Then you will have a DB. Then you will have some fancy frontend which will sure depend on node js. Then you might have some KV store/caching so you get Redis.

I completely disagree. Normally you choose ONE core language and you use only that in a monolith application. I can admit different languages for the unit tests, like groovy on java side or F# on .net. Or using two languages on the same platform, like Scala and Java on the JVM and C# + F# on .net. Or one language server side with the web side written in javascript or in something else. If you are not breaking your ap…

Even using 4 languages is too many, e.g. Java and Scala for system building, Apache Groovy for build scripts and unit testing, and Javascript for web side as in your example. Go with something like Kotlin, which compiles to JS, Android, and JVM bytecode, can be used for Gradle build scripts, and has builder-style syntax useful for test scripts. A single language is better than 3 or 4.

Re: Data Loss at GitLab

#168

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

Transparency is not a substitute for competency. Both are important, but I'll take the latter every time.

Re: Data Loss at GitLab

#169

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

Just to stress one thing, while mentioned in their report, most people don't comprehend:

Gitlab was _one_ (last!) final step from complete data loss of everything. One. At that night, there was quite a long moment there had only one copy (and 6 hours old). Every other backup was missing/notworking/deleted.

This is scary.

Re: Data Loss at GitLab

#170
post #169

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

Just to stress one thing, while mentioned in their report, most people don't comprehend: Gitlab was _one_ (last!) final step from complete data loss of everything . One. At that night, there was quite a long moment there had only one copy (and 6 hours old). Every other backup was missing/notworking/deleted. This is scary.

This is the fact that deserves more attention. The operator did a manual backup before work which was the only available one.

I find this to be an eye opener. The real problem was dodged by doing the right thing in the proverbial last minute (6 hours).

Post reply on HN