Live data from Hacker News

Data Loss at GitLab

blog.2ndquadrant.com

201–210 of 236 posts

Re: Data Loss at GitLab

#201

Earlier quoted context omitted.

I think you might be confused. One of the nice things to happen to OSS was the availability of transistors and electricity. Both made it possible to make millions of logic calculations per second without knowing it. :)

I think you might be confused. One of the nice things to happen to OSS was the availability of human beings to actually develop the concept of OSS and any underlying technology.

I think I might be confused.

Re: Data Loss at GitLab

#202

The gitlab situation and Uber's article speak to the level of immaturity of PGSQL's native replication feature, and more importantly: how not widely google-able nor documented/adopted the replication strategies are.

i chose postgres over mysql at my current job because there was an easy to use backup scripts (wal-e) available at the time and i didn't see anything comparable for mysql. so we have a hot slave and also streaming backups constantly to s3 and a full base backup every week and this with very minimal work. [https://github.com/wal-e/wal-e]

i actually regret not using mysql because mysql at the time supported out of the box logical replication which would make database upgrades easier. we haven't upgraded our postgres DB boxes because it would involve a lot of pain. whereas if we were using mysql we probably would just upgrade the slaves and wait for a failure.

Re: Data Loss at GitLab

#203
post #125

Does nobody else find the report cringeworthy? Apparently, there are some junior engineers fumbling around and committing serious errors, but where are the senior ones and the process/failsafes to prevent all this?

From what I know of the dev who caused this, he is certainly no junior engineer. One of the smartest devs I know.

Re: Data Loss at GitLab

#204
On the other hand, even if there was some data loss, shouldn't most people have their entire repositories on their drives (at least the ones they're actively working on) so in theory much can be recovered by end-users who are active. The only true worry is inactive users. Not sure if this was discussed much.

Re: Data Loss at GitLab

#205

Earlier quoted context omitted.

Yes, but if you look at the average here its not really outrageous or "5 times the average salary" as you said, here: https://www.indeed.com/salaries/Gitlab-Salaries And its not just me, they've done this in the past as well: https://news.ycombinator.com/item?id=10924957 Also when you work for a remote company, they are cutting cost in terms of office and all which should reflect back in the salaries, the whole idea…

Like I said, they're a remote company, and therefore can hire people in cheaper regions, and don't have to adapt to more expensive markets (like your own). I fail to see how this disproves it, it merely proves that you're not a good choice for them (because you live in a, relatively, expensive region).

[deleted]

Re: Data Loss at GitLab

#206

Earlier quoted context omitted.

Yes, but if you look at the average here its not really outrageous or "5 times the average salary" as you said, here: https://www.indeed.com/salaries/Gitlab-Salaries And its not just me, they've done this in the past as well: https://news.ycombinator.com/item?id=10924957 Also when you work for a remote company, they are cutting cost in terms of office and all which should reflect back in the salaries, the whole idea…

Like I said, they're a remote company, and therefore can hire people in cheaper regions, and don't have to adapt to more expensive markets (like your own). I fail to see how this disproves it, it merely proves that you're not a good choice for them (because you live in a, relatively, expensive region).

I do not see them hiring talent from cheaper countries. https://about.gitlab.com/team/

This is what is wrong with the tech world where people with not much exposure or experience mess up the numbers and remote companies like Gitlab love to take advantage of it.

Re: Data Loss at GitLab

#207

Earlier quoted context omitted.

Yes, but if you look at the average here its not really outrageous or "5 times the average salary" as you said, here: https://www.indeed.com/salaries/Gitlab-Salaries And its not just me, they've done this in the past as well: https://news.ycombinator.com/item?id=10924957 Also when you work for a remote company, they are cutting cost in terms of office and all which should reflect back in the salaries, the whole idea…

Like I said, they're a remote company, and therefore can hire people in cheaper regions, and don't have to adapt to more expensive markets (like your own). I fail to see how this disproves it, it merely proves that you're not a good choice for them (because you live in a, relatively, expensive region).

Most of their staff is from US/UK, but they just do not want to pay fair wages, which is why I posted the other thread where it was highlighted more.

Re: Data Loss at GitLab

#208

Earlier quoted context omitted.

I am sick of the "They are transparent so it's ok" argument. The fact is this type of thing should not happen anymore.. I get it, stuff fails but it's not acceptable to lose production data at this scale and save face by being transparent. That was ok back in 2003.. Not anymore.

Compare with Github and their 98.935% uptime over the past month: https://status.github.com/graphs/past_month Neither should be exclusively relied upon for business-critical services. Ask Github about their production backups and DR plans some time.

I still remember the outage where github ran their tests against production - which started by TRUNACTEing everything before loading fixtures.

I think "everybody learns the hard way" is just one of those things with operations.

Re: Data Loss at GitLab

#209

Earlier quoted context omitted.

I know (because I use it and also host it at home, and like it). The point of my comment was HN's reaction.

Remember that HN is not a single-minded mob, during each incident a particular group was vocal on a given subject but that group may not have been the same individuals who were vocal each time.

Fair point, thank you.

Re: Data Loss at GitLab

#210

Earlier quoted context omitted.

>GitLab cost everyone a day That's just nonsense. In most cases, the local repo should be more than enough to continue work.

Yeah, kind of defeats the whole D part of DVCS when you centrally-host everything. GitHub is great for creating an open-source watering hole and becoming a de-facto standard for open source project hosting, but bad for mission-critical stuff if you're relying on something like that to be 100% available, especially when using a technology which is designed to be distributed .

It would have been really cool if GitHub had fully bought into the distributed model and built the wiki/issue tracker/etc such that they were included in the repository, so that they were also decentralized.
Post reply on HN