Aside from the incident, this is a great opportunity to learn something new (at least for me). Said so, does anyone of you know how they plot all the charts of postgres and co? such as this: http://monitor.gitlab.net/dashboard/db/postgres-queries I know that the chart is made in grafana, but: how do they collect the data?
Data Loss at GitLab
161–170 of 236 posts
Re: Data Loss at GitLab
#162I don't get to work on databases this size and today has been an incredible lesson and a journey. I've been reading all the comments and blogs, watching the stream and Googling what I didn't know or understand. I feel like the next step for me is scaling my business so that we have an actual usage for my newly found interests :)
For those wondering about the live Gitlab stream, you can watch them work here: https://www.youtube.com/watch?v=nc0hPGerSd4
Re: Data Loss at GitLab
#163With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.
HN can be funny sometimes. GitHub got a lot of hate about a year ago just for not releasing new features. GitLab cost everyone a day because their backup/ops practices were silly, and everyone loves them more. I've screwed up before, and I sympathize/empathize with their ops folks, but this should make us think about plan B in case something like this happens again.
They are just better at hiding it, smoothing it over and lying.
Re: Data Loss at GitLab
#164Earlier quoted context omitted.
I think you are confused. The best thing to happen to OSS is GitHub, not GitLab. GitLab is just a fast follower and likely wouldn't even exist without the former. I for one am happy to throw money GitHub's way for their role in so dramatically changing how we code.
Github itself is not open source. The irony.
For a company that praises open-source so much they really do try to be as closed as possible themselves.
Re: Data Loss at GitLab
#165Earlier quoted context omitted.
Why? Everyone has their own copy of the gitlab remote.
We use gitlab.com for code reviews and CI, and we basically lost a day of integration productivity because of it. Yes, our process is too tightly tied to a single service, but it happens because we don't have the resources to self-host our own solution. We love GitLab, but this has absolutely got us looking elsewhere.
If you self-hosted code reviews/CI, would you expect having a similar downtime causing problem in the future?
I expect the answer to be "Yes" for most companies.
Re: Data Loss at GitLab
#166Earlier quoted context omitted.
There not many project these days that would not require similar stacks. At least one core language: be it Ruby, or Java or whatever. Then you will have a DB. Then you will have some fancy frontend which will sure depend on node js. Then you might have some KV store/caching so you get Redis.
I completely disagree. Normally you choose ONE core language and you use only that in a monolith application. I can admit different languages for the unit tests, like groovy on java side or F# on .net. Or using two languages on the same platform, like Scala and Java on the JVM and C# + F# on .net. Or one language server side with the web side written in javascript or in something else. If you are not breaking your ap…
Re: Data Loss at GitLab
#167Re: Data Loss at GitLab
#168With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.
Re: Data Loss at GitLab
#169With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.
Gitlab was _one_ (last!) final step from complete data loss of everything. One. At that night, there was quite a long moment there had only one copy (and 6 hours old). Every other backup was missing/notworking/deleted.
This is scary.
Re: Data Loss at GitLab
#170With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.
Just to stress one thing, while mentioned in their report, most people don't comprehend: Gitlab was _one_ (last!) final step from complete data loss of everything . One. At that night, there was quite a long moment there had only one copy (and 6 hours old). Every other backup was missing/notworking/deleted. This is scary.
I find this to be an eye opener. The real problem was dodged by doing the right thing in the proverbial last minute (6 hours).