Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

461–470 of 621 posts

Re: GitLab Database Incident – Live Report

#461

Post by erichocean »

blowski wrote: post #448
OskarS wrote: post #420

Earlier quoted context omitted.

Honestly, if I were interviewing the guy, that would almost be a bonus! Like, everyone makes mistakes, we're all human, but I can guarantee you that THAT person will never make that particular mistake ever again. And he's going to be 10 times more diligent than the average engineer in making sure there are good backup/restore procedures.

There's a probably apocryphal story like this about a guy forgetting to refuel a plane. The pilot made sure that guy was solely responsible for refuelling his plane in future, because he knew he'd never forget again.

I think that might have been in Chuck Yeager's (auto-?) biography. (Great read, BTW.)

Re: GitLab Database Incident – Live Report

#462

Post by koolba »

rleigh wrote: post #459
koolba wrote: post #384

Earlier quoted context omitted.

I see LVM[1] mentioned in the notes. It allows you to, among other things, snapshot a filesystem atomically which you could then mount read-only to a separate location to read for backups or export to a different environment. That would give you a point in time view of the state of all the repos that should be as consistent as a "stop the world then backup" approach. [1]: https://en.wikipedia.org/wiki/Logical_volume_…

LVM snapshots the raw block device (logical volume). The filesystem is layered on top of that, and then open and partially written files on top of that. So snapshotting an active database is really not the best idea; it might work, it should work, but it'll need to discard any dirty state from the WAL when you restart it with the snapshot. You might be in for more trouble with other data and applications, depending u…

> LVM snapshots the raw block device (logical volume). The filesystem is layered on top of that, and then open and partially written files on top of that. So snapshotting an active database is really not the best idea; it might work, it should work, but it'll need to discard any dirty state from the WAL when you restart it with the snapshot. You might be in for more trouble with other data and applications, depending upon their requirement for consistency.

> It's definitely not as consistent as "stop the world then backup" because the filesystem is dirty, and the database is dirty. It's equivalent to yanking the power cord from the back of the system, then running fsck, then replaying all the uncommitted transactions from the WAL.

I was referring to using LVM to snapshot the filesystem where the git repos are hosted. It'd work for a database as well, assuming your database correctly uses fsync/fdatasync, and for git specifically it works fine.

Using LVM snapshots with a journaled filesystem (i.e. any modern/sane choice for a fs) should have no issues though there would be some journal replay at mount time to get things consistent (v.s. say ZFS which wouldn't require it). If it does have issues, you'd have the same issues with the raw device in the event of hard shutdown (ex: power failure).

Re: GitLab Database Incident – Live Report

#463

Post by spudlyo »

jfindley wrote: post #350

"The replication procedure is super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented" This is true of many databases, but in my experience is particularly true of postgres. It's a marvelous single-instance product, but I've never really found any replication/ha tech for it that I've been that happy with. I've always been a bit nervous about postgres-backed products for thi…

I've been a DBA for thousands of MySQL hosts in production and just a handful of PostgreSQL clusters, so I'm not a PostgreSQL expert by any stretch. These PostgreSQL machines were also inherited from a company we acquired, so some of my negative impressions can likely be explained by them being set up poorly, which sadly seems to be the norm. PostgreSQL's replication has evolved slowly over the years, and as a result it's idiosyncratic, complicated to get right, and hard to generalize about.

Postgres replication works by shipping WAL (transaction/redo) logs to slaves, and the slaves are in a constant state of DB recovery. Streaming replication does so as writes happen, and file based log shipping copies the logs when the 16MB segment is complete. Ours were set up to use streaming replication, without having a separate file-based archive log host to use to fetch older logs. Whenever the write load became too much and the slaves could no longer keep up with the master WAL logs on the master would expire and replication would break entirely, requiring you to rebuild all your slaves from backup. This was my introduction to PG replication. Apparently PG 9.4 has 'replication slots' or something that make this less of a headache.

WAL logs were also applied without checksums or verification which corrupted the entire replication chain, and as I remember we didn't discover this until the corrupt pages got hit by a query.

Software upgrades with PG seemed to be a huge pain as well, we could never figure out how long they were going to take, and even if the on-disk formats didn't change, somehow all the cardinality stats on our tables disappeared, and all our query plans went to shit until we ran some process to rebuild them, which took days.

Operationally there were some things that made our team angry. We couldn't figure out how to reparent a slave without completely re-cloning it from the new parent, even though it was completely up to date from the authoritative master at the time of the reparenting. Also, do you really have to take down the entire cluster to change max connections on a slave? Many such settings seemed to not be dynamic and must remain in sync across the entire chain. One of the things people seem to love about PG is how correct and proper it is respecting the sanctity of your data. As an ops person I'd much rather deal with slightly inconsistent replicas (common in MySQL) than have to fight with how rigid PG is.

Re: GitLab Database Incident – Live Report

#465

Post by traviscj »

hobarrera wrote: post #136
shawn-furyan wrote: post #123

Earlier quoted context omitted.

Or even instead of any kind of rm command. mv is less subtle. I tend to prefer `mv x $(date +%Y%m%d_%s_)x` where: %Y - 4 digit year %m - 2 digit month %d - 2 digit day _ - underscore literal %s - linux timestamp (seconds since epoch) This ensures that the versions you're 'removing' will be lexically sorted from newest to oldest in a way that is easy to interpret and also works if you need to try more than once in a d…

date -Im is shorter and easier to remember (though not 100% standard, IIRC). ;) Example: 2006-08-14T02:34:56-06:00

You can "brew install coreutils" and then use "gdate " on macOS.

Re: GitLab Database Incident – Live Report

#466

Post by fbreduc »

Traubenfuchs wrote: post #392

So tl;dr: Gitlab is experiencing heavy DOS attacks that created so much data that replication stopped working. In the process of getting replication to work again, "YP" wanted to delete the empty data directory of the slave DB server, but accidentally deleted it on the master DB server. Out of 5 backup/replication techniques they use not one is working reliably. YP manually created the backup they could use 6 hours a…

>Tell me if I misunderstood something. I hope the customer I met last week does not remember I ever recommended GitLab to him.

this is how i feel right now, i suggested gitlab to half a dozen people who eventually moved to it for their business work... now i'm such an asshole :(

Re: GitLab Database Incident – Live Report

#467

Post by traviscj »

wtbob wrote: post #165
eric_h wrote: post #14

Earlier quoted context omitted.

Good lesson on making command prompts on machines always tell you exactly what machine you're working on.

'db1' vs. 'db2' is still insufficiently clear, though. Even better would be e.g. to name development systems after planets and production systems after superheroes. Very few people would mistake 'superman' for either 'green-lantern' or 'pluto,' but it's really easy to mistake 'sfnypudb13' for 'sfnydudb13.'

At work we have an ascii-art, most-of-80x40-filling version of our logo rendered in an environment-associated color on login + matching prompt. The ascii-art logo might not have helped in this case (if it was a long sequence of console interactions), but It does catch "db1" vs "db2" typos, for instance, and also elicits a certain reverence upon connection.

Re: GitLab Database Incident – Live Report

#468

Post by my_ghola »

CrLf wrote: post #415
nanch wrote: post #413

Earlier quoted context omitted.

Anybody whose opinion matters understands that this type of event is a process problem, not a person problem. GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names). To be honest, through all of the comments, yours seems the mo…

In a few years the guy doing the `rm -rf` is going to be on a job interview and someone will recall bits of this report. Enough bits to remember the guy, not enough bits to remember that it wasn't his (individual) fault. Transparency doesn't mean publicly throwing people under the bus. I'm not a GitLab customer, I'm relaxed. :)

I'd rather hire someone who learned what not to do than someone who hasn't yet.

Re: GitLab Database Incident – Live Report

#469

Post by Ajedi32 »

CrLf wrote: post #405

The public report is nice and we can see a sequence of mishaps from it, that shouldn't have been allowed to happen but which (unfortunately) are not that uncommon. I've done my share of mistakes, I know what's like to be in emergency mode and too tired to think straight, so I'm going to refrain from criticizing individual actions. What I'm going to criticize is the excess of transparency: You absolutely DO NOT publis…

Am I missing something? Where in this report are any individuals actually named? My understanding was that they're using initials in place of names specifically because they want to _avoid_ naming anyone.

Re: GitLab Database Incident – Live Report

#470

Post by ausjke »

That's why I do not use github/gitlab/whatever to host the part of my code that is too critical to me. I push it to my ssh/git server and use local UI to interact with it instead.

Sometimes source code is very valuable and you just can not make any mistakes with it.

Post reply on HN