Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Loss at GitLab

blog.2ndquadrant.com

Post reply on HN
Print view View original

221–230 of 236 posts

Re: Data Loss at GitLab

#221

Post by dpc_pw »

anonymous_green wrote: post #178
aamederen wrote: post #29

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

Transparency? Gitlab is the company that interviews people and rejects them on basis of salary, they've been doing this for awhile, every few months they call a group of people, waste their time and then deny based on the salary. In my case, I've confronted them that USD 100-120k is the market average but they had this stupid startup argument that doesn't make any sense to me.

What's wrong with that? Isn't it obvious that salary is an important factor in hiring people?

Re: Data Loss at GitLab

#222

Post by corobo »

dijit wrote: post #191
sytse wrote: post #159

Earlier quoted context omitted.

I totally agree with needing an experienced DBA, we've had a vacancy open for this for a while https://about.gitlab.com/jobs/specialist/database/

They want a developer who can do database stuff, not a DBA (based on that job posting.)

They are who you're replying to

> I'm the CEO of GitLab https://about.gitlab.com/ More information about me is on http://sytse.com

Re: Data Loss at GitLab

#223

Post by dijit »

corobo wrote: post #222
dijit wrote: post #191

Earlier quoted context omitted.

They want a developer who can do database stuff, not a DBA (based on that job posting.)

They are who you're replying to > I'm the CEO of GitLab https://about.gitlab.com/ More information about me is on http://sytse.com

Fair enough, but I don't see him denying it.

Re: Data Loss at GitLab

#224

Post by developer2 »

sytse wrote: post #159
developer2 wrote: post #137

Earlier quoted context omitted.

I agree. Their catastrophe is not the kind of thing that represents mild hiccups in operations that will be quickly resolved by the same employees who allowed this scenario to occur the first time around. This isn't the result of a single oversight. It screams of a systemic problem with the way the business operates - period. Maybe every project is rushed out with the deadline being the only metric that counts, quali…

I totally agree with needing an experienced DBA, we've had a vacancy open for this for a while https://about.gitlab.com/jobs/specialist/database/

>> As a database specialist not only will you work on the database, but also the application's usage of the database. As a result a significant amount of experience with both Ruby and PostgreSQL is a strict requirement.

This is a mistake. The number of DBAs who are good with postgres is very small compared to something like mysql. The truly talented pool for such a position is too small to expect them to also be a developer. The very mention of terms like "ruby" and "programming" should be removed from that job post. It's not a realistic expectation.

Re: Data Loss at GitLab

#225

Post by spinlock »

aamederen wrote: post #29

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

For me it underscores one simple rule: you backups are useless if you don't regularly practice restoring from them.

Re: Data Loss at GitLab

#226

Post by bartvk »

hobarrera wrote: post #189
anonymous_green wrote: post #186

Earlier quoted context omitted.

Yes, but if you look at the average here its not really outrageous or "5 times the average salary" as you said, here: https://www.indeed.com/salaries/Gitlab-Salaries And its not just me, they've done this in the past as well: https://news.ycombinator.com/item?id=10924957 Also when you work for a remote company, they are cutting cost in terms of office and all which should reflect back in the salaries, the whole idea…

Like I said, they're a remote company, and therefore can hire people in cheaper regions, and don't have to adapt to more expensive markets (like your own). I fail to see how this disproves it, it merely proves that you're not a good choice for them (because you live in a, relatively, expensive region).

> because you live in a, relatively, expensive region

If what OP is saying is true, then the company might as well be upfront about it.

Re: Data Loss at GitLab

#227

Post by haylem »

briffle wrote: post #195
sytse wrote: post #159

Earlier quoted context omitted.

I totally agree with needing an experienced DBA, we've had a vacancy open for this for a while https://about.gitlab.com/jobs/specialist/database/

Just as an aside, you can hire a remote DBA to help out, until you get a full time person. We have a contracted 3rd party that does secondary monitoring around the clock, and a very, very good pro that we work with on projects. In our case, we have a 'retainer' for 15 hours a month. (and a fixed cost per hour after)

The entire GitLab team is remote, as far as I understand their company model.

https://about.gitlab.com/2015/04/08/the-remote-manifesto/

But maybe you meant part-time?

Re: Data Loss at GitLab

#228

Post by xenithorb »

spinlock wrote: post #225
aamederen wrote: post #29

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

For me it underscores one simple rule: you backups are useless if you don't regularly practice restoring from them.

Or I don't know, at least checking they exist

Re: Data Loss at GitLab

#229

Post by i336_ »

Girlang wrote: post #94
mi100hael wrote: post #38

Earlier quoted context omitted.

It provides a lot of the same features as GitHub, only you can self-host for free, contribute to development, and all the other niceties of a free project. A lot of people (myself included, and OP I suspect) believe it's also a particularly good project because GitHub has too much market share and influence in the free software ecosystem for a for-profit company.

Plus github has an agenda that has nothing to do with code, and excludes large segments of the public.

Your comment was killed (I just vouched for it) because of your unsubstantiated claims. Can you give some concrete examples of why you think this way?

Re: Data Loss at GitLab

#230

Post by developer2 »

sytse wrote: post #159
developer2 wrote: post #137

Earlier quoted context omitted.

I agree. Their catastrophe is not the kind of thing that represents mild hiccups in operations that will be quickly resolved by the same employees who allowed this scenario to occur the first time around. This isn't the result of a single oversight. It screams of a systemic problem with the way the business operates - period. Maybe every project is rushed out with the deadline being the only metric that counts, quali…

I totally agree with needing an experienced DBA, we've had a vacancy open for this for a while https://about.gitlab.com/jobs/specialist/database/

To be a bit more specific, it is extremely unlikely (nearly impossible) that you could ever find a single individual who is both a "postgres database specialist" and a "ruby developer". Do you even know what the word "specialist" means? It means specializing in one area - like postgres - without trying to be a Jack of all trades, ie: with ruby.

"DBA" is a full-time position, not an addon to a developer's duties. They are separate skillsets; you don't get a "2-for-1" special by hiring an expert developer + DBA in one person for one lowly salary.

Do you realize this event would have never happened if you had hired a pure DBA? Or do you really believe you can pin the blame on your ruby developers for not being able to wrangle a production postgres database?

The oblivious or intentionally cheap "the DBA must be an amazing ruby developer" expectation shows your hiring staff - or the management guiding them - has absolutely no clue what they are doing. I can just imagine the internal discussion right now; pointing the finger at the developers with no postgres experience, or downplaying the significance of this event and pretending like it was simply bad luck, and lying to yourselves about how "it will never happen again".

This job posting is completely outside the realm of reason. If that job posting has been up for months or years, I can see its description being exactly why you didn't have the right talent on board to avoid this incident.

Post reply on HN