Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Loss at GitLab

blog.2ndquadrant.com

Post reply on HN
Print view View original

191–200 of 236 posts

Re: Data Loss at GitLab

#191

Post by dijit »

sytse wrote: post #159
developer2 wrote: post #137

Earlier quoted context omitted.

I agree. Their catastrophe is not the kind of thing that represents mild hiccups in operations that will be quickly resolved by the same employees who allowed this scenario to occur the first time around. This isn't the result of a single oversight. It screams of a systemic problem with the way the business operates - period. Maybe every project is rushed out with the deadline being the only metric that counts, quali…

I totally agree with needing an experienced DBA, we've had a vacancy open for this for a while https://about.gitlab.com/jobs/specialist/database/

They want a developer who can do database stuff, not a DBA (based on that job posting.)

Re: Data Loss at GitLab

#193

Post by Ajedi32 »

taneq wrote: post #144
BHSPitMonkey wrote: post #67

Earlier quoted context omitted.

Not everyone has their own copy of the project's issues/pull requests/settings/webhooks/CI config/etc., which the incident affected (and 6 hours worth were lost). Git repositories themselves do have the happy side effect that at least someone on your team has a full latest clone on their machine, but that's only because the source code is what you're there to use every day. Even if GitLab had a great "checkout your i…

Maybe a dumb question but couldn't they be included in a hidden folder in the root of your repo or something? That way they'd get replicated automatically every time someone does a pull.

It'd probably have to be on a separate branch, but yeah they technically _could_ do that. Maybe something like this? https://github.com/duplys/git-issues

Re: Data Loss at GitLab

#194

Post by empath75 »

synicalx wrote: post #104

Honestly, I'm completely flabbergasted by this. Five backups, and NONE worked properly? Who made this? The S3 bucket was EMPTY? Has no one ever tested any of these backups? It's not just the impact, which is fairly sizeable in it's own right, but it's the HUGE oversight on their part and the fact they tried to pin part of this on PostreSQL? Credit where it's due; their report/transparency were good if a little unprof…

I got thrown on to the team managing netbackup at a massive company a few years ago, and it's actually fairly common that people don't test backups. We sent out notifications constantly warning people that if they aren't testing their backups, they aren't really backing up, but at least once every other month, someone would come back to us wondering why their backups weren't there, when they were trying to recover something.

Re: Data Loss at GitLab

#195

Post by briffle »

sytse wrote: post #159
developer2 wrote: post #137

Earlier quoted context omitted.

I agree. Their catastrophe is not the kind of thing that represents mild hiccups in operations that will be quickly resolved by the same employees who allowed this scenario to occur the first time around. This isn't the result of a single oversight. It screams of a systemic problem with the way the business operates - period. Maybe every project is rushed out with the deadline being the only metric that counts, quali…

I totally agree with needing an experienced DBA, we've had a vacancy open for this for a while https://about.gitlab.com/jobs/specialist/database/

Just as an aside, you can hire a remote DBA to help out, until you get a full time person. We have a contracted 3rd party that does secondary monitoring around the clock, and a very, very good pro that we work with on projects. In our case, we have a 'retainer' for 15 hours a month. (and a fixed cost per hour after)

Re: Data Loss at GitLab

#197

Post by throwaway2048 »

bbcbasic wrote: post #181
juskrey wrote: post #122

Earlier quoted context omitted.

This is basic risk management. What matters is error survival, not absence of errors. Moreover, total absence of errors means accumulating dangerous risk. See NN Taleb, for example.

Sounds defeatist to me. Especially when we are not talking about literal moonshots, but backup and restore procedures. The IT equivalent of seatbelts.

I think at this point, recognising that software and software deployments invariably have bugs/issues is just realistic. The only setups that are free of them, are setups where the problems havent been discovered yet.

Re: Data Loss at GitLab

#198

Post by justinclift »

BrentOzar wrote: post #190
sytse wrote: post #159

Earlier quoted context omitted.

I totally agree with needing an experienced DBA, we've had a vacancy open for this for a while https://about.gitlab.com/jobs/specialist/database/

The word "backup" doesn't appear anywhere in the job description. Might wanna revise that today.

They'd probably be better off hiring a "backup engineer" to go through all of their systems (end to end) ensuring full backups exist and work for everything.

Instead of backing up every system in isolation, have well though out backup/restore processes for all parts of their operation.

Saying that as someone who's done exactly this before (professionally, for mission critical places). ;)

eg:

· Inventory the systems (boxes, services, etc)

· Determine what each needs (package dependencies, etc)

· Create scripting (etc) for consistent backups

· Work out the restore processes

· Make it work (can take several test/dev iterations)

· Document it

And also (importantly):

· Have the ops staff perform the documented processes, to reveal holes in the docs, and show up parts which need simplifying

Re: Data Loss at GitLab

#199

Post by ysavir »

shakna wrote: post #184
mondoshawan wrote: post #183

Earlier quoted context omitted.

The one with the CVEs because it has had eyes on the source enough to generate them. The devil you know and all that.

Exactly. They've made mistakes, and can recover from them. The other hasn't had that chance. Thus, try the first, as it is better tested in the world.

That's very different, though. I trust the company/organization that had to fire an employee for a horrible mistake. They are going to be very careful with who they hire or let near the code in the future, and had much more to lose than a single person.

But I don't trust the employee that was shown forgiveness for a horrible mistake. Some might learn to not repeat what they had done, while others learn that they can get away with things through the magical power of phrasing the situation in a positive light. And some might be mistake-makes-for-life. Not all people come out the other side stronger.

Re: Data Loss at GitLab

#200

Post by s_kilk »

crististm wrote: post #170
rdslw wrote: post #169

Earlier quoted context omitted.

Just to stress one thing, while mentioned in their report, most people don't comprehend: Gitlab was _one_ (last!) final step from complete data loss of everything . One. At that night, there was quite a long moment there had only one copy (and 6 hours old). Every other backup was missing/notworking/deleted. This is scary.

This is the fact that deserves more attention. The operator did a manual backup before work which was the only available one. I find this to be an eye opener. The real problem was dodged by doing the right thing in the proverbial last minute (6 hours).

In other words, there is an alternate universe in which Gitlab is now categorically dead, all because that employee didn't make that single backup.
Post reply on HN