Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

511–520 of 621 posts

Re: GitLab Database Incident – Live Report

#511

Post by ebbv »

nsxwolf wrote: post #503
ebbv wrote: post #453

Earlier quoted context omitted.

When we interview people one of the questions we like to ask is "What's the biggest thing you've accidentally deleted?" When people answer that question honestly and with humility it is a big plus.

Might be a plus for your organization, and it might be devised as a trap by another. It's like the biggest weakness question. Oh, we can't hire someone who has made a mistake THAT big.

You don't want to work for a company that has that attitude anyway, honestly. That shows they have a poor attitude towards problems and probably will overreact to things like missing deadlines or pursuing a solution that ends up not working, etc.

Re: GitLab Database Incident – Live Report

#512

Post by testUser69 »

YorickPeterse wrote: post #476
CrLf wrote: post #405

The public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncommon. I've done my share of mistakes, I know what's like to be in emergency mode and too tired to think straight, so I'm going to refrain from criticizing individual actions. What I'm going to criticize is the excess of transparency: You absolutely DO NOT publis…

I myself initially added my name to the document in various parts, this was later changed to just initials. I specifically told my colleagues it was OK to keep it in the document. I have no problems taking responsibility for mistakes, and making sure they don't happen ever again.

I don't know why people on hacker news are against transparency. I'm glad you guys are live streaming this, others would feel too inadequate to do so. Being this transparent only makes me want to use (and contribute to) gitlab even more.

Re: GitLab Database Incident – Live Report

#513

Post by nsxwolf »

ebbv wrote: post #511
nsxwolf wrote: post #503

Earlier quoted context omitted.

Might be a plus for your organization, and it might be devised as a trap by another. It's like the biggest weakness question. Oh, we can't hire someone who has made a mistake THAT big.

You don't want to work for a company that has that attitude anyway, honestly. That shows they have a poor attitude towards problems and probably will overreact to things like missing deadlines or pursuing a solution that ends up not working, etc.

You'll never know if it was a great company with a bad interviewer. It's better to use all the advantages you can to get through an interview and get the job to see for yourself. I don't think you can learn anything definitive from most interviews - they're mostly subjective, unscientific voodoo.

Re: GitLab Database Incident – Live Report

#515

Post by d3ad1ysp0rk »

throwaway3347 wrote: post #299
ahawkins wrote: post #265

Earlier quoted context omitted.

> They really should focus on creating jobs that pays $200,000+ a year, regardless of living location, to attract the best talent from around the world. Getting 3-6 top talent, that can help steer the company in the right direction, can make all the difference in the long run. SIGN ME UP! That would be a freaking great opportunity!!

> SIGN ME UP! That would be a freaking great opportunity!! I think you asking for the job, might be a signal, that you are not who they are looking for :-)

Yup - top talent is already making more. Gitlab needs to recruit with purpose (this is what we're doing and why), environment (remote first, transparency, etc), and pay (we can match 70% of what you'd get at XYZ Company). Right now, it feels like they're capped at 30-50% of what someone could make at a big org, which is just a drop in salary most people would never take, regardless of the company values/purpose.

One alternate idea would be to hire consultants on a temporary basis. You may not be able to pay $250k a year, but you could pay a one time $40k fee to review the architecture and come up with prioritized strategy for disaster recovery and scalability.

Re: GitLab Database Incident – Live Report

#516

Post by Tephlon »

testUser69 wrote: post #512
YorickPeterse wrote: post #476

Earlier quoted context omitted.

I myself initially added my name to the document in various parts, this was later changed to just initials. I specifically told my colleagues it was OK to keep it in the document. I have no problems taking responsibility for mistakes, and making sure they don't happen ever again.

I don't know why people on hacker news are against transparency. I'm glad you guys are live streaming this, others would feel too inadequate to do so. Being this transparent only makes me want to use (and contribute to) gitlab even more.

I'm guessing they feel strongly about getting singled out if something like this would happen to them. Possibly because they have been used as a scapegoat by a employer or team mate once.

/ArmchairPsychology

Re: GitLab Database Incident – Live Report

#517

Post by Bad_CRC »

theptip wrote: post #18
ams6110 wrote: post #9

23:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.

Also, as a safety net, sometimes you don't need to run `rm -rf` (a command which should always be prefaced with 5 minutes of contemplation on a production system). In this case, `rmdir` would have been much safer, as it errors on non-empty directories.

lsof -> check if a process is accessing that file/directory.

Re: GitLab Database Incident – Live Report

#518

Post by neurotech1 »

PKop wrote: post #307
sschueller wrote: post #282

Earlier quoted context omitted.

There is a neat video[1] where a Swiss flight has to make an emergency landing and just happens to have a film crew in the cockpit. [1] https://www.youtube.com/watch?v=rEf35NtlBLg

Here's a great documentary [0] by Errol Morris about the United Flight 232 crash in 1989 [1]. "..the accident is considered a prime example of successful crew resource management due to the large number of survivors and the manner in which the flight crew handled the emergency and landed the airplane without conventional control." I highly recommend it [0] https://www.youtube.com/watch?v=2M9TQs-fQR0 [1] https://en.wi…

Another recent example is Qantas QF32 had an engine explode (fire then catastrophic/uncontained turbine failure) and the A380 landed with one good engine, and two degraded engines. The entire cockpit crew of 5 pilots did a brilliant job in landing the jet.

Re: GitLab Database Incident – Live Report

#519

Post by Tephlon »

blowski wrote: post #448
OskarS wrote: post #420

Earlier quoted context omitted.

Honestly, if I were interviewing the guy, that would almost be a bonus! Like, everyone makes mistakes, we're all human, but I can guarantee you that THAT person will never make that particular mistake ever again. And he's going to be 10 times more diligent than the average engineer in making sure there are good backup/restore procedures.

There's a probably apocryphal story like this about a guy forgetting to refuel a plane. The pilot made sure that guy was solely responsible for refuelling his plane in future, because he knew he'd never forget again.

Exactly.

I run backups on my computer before installing new software/fiddling with important settings/etc. because I've fucked up before.

I'll run backups of phones (or at least verify that they are present) before trying to fix issues on them after nuking my mom's phone which resulted in her losing pictures of my niece and nephew. (Luckily she had sent a lot of those pictures to us via e-mail, but still).

We learn and adjust.

Re: GitLab Database Incident – Live Report

#520

Post by luckystartup »

debaserab2 wrote: post #258
iagooar wrote: post #245

Earlier quoted context omitted.

I'm a bit curious here. Do you think that your issues with scalability and reliability have to do with your tech choice (I think it was Ruby on Rails)? Don't want to bash Rails, I'm just genuinely curious, since I come from a Rails background as well and have seen issues similar to yours in the past.

You can check out their codebase here: https://github.com/gitlabhq/gitlabhq Just looking at their gemfile is rather telling: a couple hundred gems. I've always felt that if you're going above 100, you should carefully consider how much your codebase is trying to achieve. They're probably at the point where they really want to think about splitting off of their monolith codebase and into microservices.

Maybe it's because I'm familiar with almost all of the gems, but I don't see anything wrong with their Gemfile. It's a pretty complex project, and they really do have a ton of integrations and features that need those gems.

There's probably a few small libraries that they could have rewritten in a few files (never a few lines), but what's the point? The version is locked, and code can always be forked if they need to make changes (or contribute fixes).

Post reply on HN