Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

281–290 of 621 posts

Re: GitLab Database Incident – Live Report

#281

Post by oblio »

hueving wrote: post #230
throwaway3347 wrote: post #209

Earlier quoted context omitted.

Like you, I would like to add my 2 cent, which I hope will be taken positively, as I would like to see them provide healthy competition for GitHub for years to come. Since GitLab is so transparent about everything, from their marketing/sales/feature proposals/technical issues/etc., they make it glaringly obvious, from time to time, that they lack very fundamental core skills, to do things right/well. In my opinion, t…

Why would they try to recruit from Microsoft? Most of the software engineers at Microsoft are not focused on developing scalable web services architectures. And the ones that do have built up all of their expertise with Microsoft technologies (.net running on Windows server talking to mssql). >Microsoft and others that have top talent/domain expertise, will have the advantage. Again, Microsoft isn't even in this same…

Hotmail, MSN, Skype, msdn.com, microsoft.com, the Windows Update Servers, Azure.

Microsoft has a ton of people with experience in building cloud system, either in-house people or people from aquisitions.

Microsoft has so many employees and domains of activity that you can probably find an engineer for any domain you're looking for.

Re: GitLab Database Incident – Live Report

#282

Post by sschueller »

ryandrake wrote: post #185
gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

If you get the chance to observe pilots operating in the cockpit, I'd recommend it. Every important procedure (even though the pilot has it memorized) is done with a checklist. Important actions are verbally announced and confirmed: "You have the controls" "I have the controls". Much of flight training deals with situational awareness and eliminating distractions in the cockpit. Crew Resource Management[1]. 1: https:…

There is a neat video[1] where a Swiss flight has to make an emergency landing and just happens to have a film crew in the cockpit.

[1] https://www.youtube.com/watch?v=rEf35NtlBLg

Re: GitLab Database Incident – Live Report

#283

Post by danielsamuels »

kelnos wrote: post #227
bpchaps wrote: post #178

Earlier quoted context omitted.

I agree with what you're saying, and this is almost exactly what I do, but when disk space is limited - particularly during time-sensitive situations - this advice isn't very useful. For example, if a host or service is on the cusp of crashing because of a partition quickly filling up with rolling logs, what do you do since mv doesn't actually solve the problem? At some level you have to run an rm, and you better hop…

You shouldn't let yourself get to that point. Your alerting system should alert you when disk is at 70% or something with a ton of margin. If it's not set up that way, go stop what you're doing and fix that. (Seriously.) If you're running your systems so they usually run at 90% disk usage, go give them more disk (or rotate logs sooner). And even assuming all that fails and I'm in that situation where I have seconds u…

> You shouldn't let yourself get to that point. Your alerting system should alert you when disk is at 70% or something with a ton of margin.

What would your recommendation be?

Re: GitLab Database Incident – Live Report

#284

Post by wingerlang »

click170 wrote: post #75
gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

BBC's Horizon has a really good episode about checklists and how they're used to prevent mistakes in hospitals, and how they're being adopted in other environments in light of that success. It's called How To Avoid Mistakes In Surgery for the interested.

Something related, I made a script that helps me to clean up my git repository of already-merged branches (I tend to not delete them until after a release cycle).

In this script I added a checklist of things to "check" before running it. It has worked in my favour every time I run it.

Here's how it looks like https://github.com/jontelang/Various-scripts/blob/master/Git...

Re: GitLab Database Incident – Live Report

#285

Post by x0x0 »

tokenizerrr wrote: post #33
m_fam_wa_k wrote: post #21

Earlier quoted context omitted.

I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.

How exactly do you color code it?

for example, in .bashrc on osx:

   RED=$(tput setaf 1)
   NORMAL=$(tput sgr0)
   PS1="\[${RED}\]PROD \[${NORMAL}\]\W\$ "
produces prompt

PROD ~$ <-- prod in red, directory ~

Re: GitLab Database Incident – Live Report

#286

Post by terom »

jorblumesea wrote: post #263

So, maybe a stupid question from a frontend dev who doesn't deal with these systems at all, but aren't these systems usually part of a cluster with read replicas? Blowing away the contents of one box shouldn't destroy the cluster right? I thought the primary/secondary pattern was really common among relational databases and failover boxes and other measures were standard practice. Was the command executed on all mach…

Replication systems can break. They work working on fixing the broken db2 read replica when they accidentially nuked the primary db1 server.

> db2.cluster refuses to replicate, /var/opt/gitlab/postgresql/data is wiped to ensure a clean replication

> [...] decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com

That ops fail on db1 wouldn't have been such an issue if they weren't in such a vulnerable position with an invalidated read replica.

Re: GitLab Database Incident – Live Report

#287

Post by x0x0 »

elementalest wrote: post #88

>1. LVM snapshots are by default only taken once every 24 hours. YP happened to run one manually about 6 hours prior to the outage >2. Regular backups seem to also only be taken once per 24 hours, though YP has not yet been able to figure out where they are stored. According to JN these don’t appear to be working, producing files only a few bytes in size. >3. SH: It looks like pg_dump may be failing because PostgreSQ…

if #2 is correct, holy shit did gitlab get lucky someone snapshotted 6 hours before.

Dear you: it's not a backup until you've (1) backed up, (2) pushed to external media / s3; (3) redownloaded and verified the checksum; (4) restored back into a throwaway; (5) verified whatever is supposed to be there is, in fact, there, and (6) alerted if anything went wrong. Lots of people say this, and it's because the people saying this, me included, learned the hard way. You can shortcut the really painful learning process by scripting the above.

Re: GitLab Database Incident – Live Report

#288

Post by foota »

atmosx wrote: post #274
hobarrera wrote: post #134

Earlier quoted context omitted.

On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.

Maybe add a bash function to check the path and ask the magic question: "Do you really want to delete the dir XYZ in root@domain.com?" .... but then again when you're in panic mode you might either misread the host or hit 'y' without really reading what's in front of you. The best thing to do is to never operate with 2 terminals simultaneously, when one of them is a production env, better login/logout or at least min…

Perhaps, Do you really want to delete the dir XYZ with 300000000000 bytes in it?

Re: GitLab Database Incident – Live Report

#289

Post by saosebastiao »

cnst wrote: post #161
gizmo wrote: post #120

Earlier quoted context omitted.

What if I told you all of society is held together by duct tape? If you're surprised that startups cut corners you're in for a rude awakening. I'm frequently amazed anything works at all.

Startups only, you say? Everything, everywhere, is held together by ducttape!

True. I watched a 60 year old manufacturing plant shut down for 7 days once because someone saved an excel spreadsheet in the wrong format and it deleted a macro written more than a decade prior that held together all of their operations planning.

Re: GitLab Database Incident – Live Report

#290

Post by perlgeek »

>2. Regular backups seem to also only be taken once per 24 hours, though YP has not yet been able to figure out where they are stored. According to JN these don’t appear to be working, producing files only a few bytes in size.

>3. SH: It looks like pg_dump may be failing because PostgreSQL 9.2 binaries are being run instead of 9.6 binaries. This happens because omnibus only uses Pg 9.6 if data/PG_VERSION is set to 9.6, but on workers this file does not exist. As a result it defaults to 9.2, failing silently. No SQL dumps were made as a result. Fog gem may have cleaned out older backups.

>5. Our backups to S3 apparently don’t work either: the bucket is empty

I think we've all seen that with some kind of report or backup or that's regularly reported: an empty file or none at all is generated due to some silent error.

I highly recommend creating a monitoring check for file size for each automatically generated file.

At $work, we also generate quite a few config files (for Radius, DHCP servers, web servers, mail servers....). For those we have a mechanism that diffs the old and the new version, and rejects the new version if the diff exceeds a pre-defined percentage of the file size, escalating the decision to a human.

Post reply on HN