Most destructive troll ever.
Glad you read it till the end.
Postmortem of database outage of January 31
141–150 of 269 posts
Re: Postmortem of database outage of January 31
#142Earlier quoted context omitted.
this is the reason why I always use quotes before specifying folders with rf -rf :)
I always type ls /stuff/wherever/* Then examine the output to see if that is the stuff I really want to delete and if it is, up-arrow Ctrl-A right right backspace backspace rm -rf enter Never deleted the wrong stuff again in 30 years of doing that
I always cd to the parent directory, ls with tab complete to check, the rm with the same tab complete.
Re: Postmortem of database outage of January 31
#143Earlier quoted context omitted.
doesn't everyone alias rm to rm -i on prod? likewise all tty's have red backgrounds on prod.
I'm not sure if you're making an equally sarcastic point as your parent or not..
nope serious. I was at a place that the dba did exactly what happened at gitlab but in sql
select * from table > script
@script.
(drop all the tables)
It was in prod, he thought it was a dev db, the backups had never worked. After this the edict was all terminals for prod will be red. A simple solution
Re: Postmortem of database outage of January 31
#144Earlier quoted context omitted.
My worst data loss was: $ tar cvfz mbox outbox mbox.tar.gz The argument order is backwards -- the output file is supposed to be first, then the input files. On my system, this overwrote my full mailbox with a gzipped copy of my outbox, and a complaint that the mbox.tar.gz input file didn't exist. That's right, the worst data loss happened while I was trying to take a backup. :(
What were you trying to do? What's the outbox argument for?
He was trying to backup his mbox and outbox.
Re: Postmortem of database outage of January 31
#145Earlier quoted context omitted.
RDS also comes with its own set of tradeoffs. There is no free lunch, and the cloud is just another word for someone else's server. There are reasons Gitlab opposes that.
In the meantime solution architects and sales people from AWS are going to run around with annotated copies of this public post-mortem to enterprises and say "look, RDS would have solved x,y,z and we can do that for you if you pay us" :)
[deleted]
Re: Postmortem of database outage of January 31
#146Earlier quoted context omitted.
It's a bit odd that living in Tokyo you get 90k. While living in SF you get 220k. I'm fairly certain the rent difference isn't 10k per month.
Supply and demand
Of what? A developer in that location? What is that benefit?
Having worked on both coasts, I can say that the quality is the same. Culture and quantity appears to be the biggest difference.
Re: Postmortem of database outage of January 31
#147Earlier quoted context omitted.
RDS also comes with its own set of tradeoffs. There is no free lunch, and the cloud is just another word for someone else's server. There are reasons Gitlab opposes that.
>> the cloud is just another word for someone else's server. No. The cloud (AWS, GCE, Azure etc) is not "just" like your own server. Just consider some basic details - you pay someone else to worry about things like power outages, disk failures, network issues, other hardware failures, and so on.
I think that's a little pedantic. The point he was making is that, conceptually speaking, the cloud is comprised of servers not unlike the servers you run yourself. The difference, obviously, is who runs them, the manner in which they're run, the exact manner in which they're utilized by you, etc., but they are still just servers at the bottom of the stack.
Re: Postmortem of database outage of January 31
#148>Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. I could feel the sweat drops just from readin…
I'm no sysadmin, and I know mistakes are inevitable and all... but I find this kind of mistake is unlikely to come from me. I feel as though a lot of developers are too nonchalant about production boxes. I think one or two close calls where I nearly did this exact thing served as a good wakeup call for me. Steps I personally take to avoid this: - Avoid prod boxes like the plague - Set up a prompt (globally) to make i…
I don't think most of your points really apply though. They were setting up replication in production, so they had to work on production boxes. Setting prompt to say just "production" wouldn't help for the same reason. Production was intended.
Peer review though - yes. That could help. I wouldn't say "I'm unlikely to make that mistake" - it's likely to go on the famous last words list...
Re: Postmortem of database outage of January 31
#149If you want to up your cron job monitoring game there's a link in my profile.
Re: Postmortem of database outage of January 31
#150Earlier quoted context omitted.
Snapshots are not backups, although many people use them as backups and believe they are good backups. Snapshots are snapshots. Only backups are backups.
What is the difference exactly?
A snapshot could be a backup depending on what you're calling a snapshot, but yeah, in general, to be a backup things need to have these features:
1. stored on separate infrastructure so that obliteration of the primary infrastructure (AWS account locked out for non-payment, password gets stolen and everything gets deleted, datacenter gets eaten by a sinkhole, etc.) doesn't destroy the data.
2. offline, read-only. This is where most people get confused.
Backups are unequivocally NOT a live mirror like RAID 1, slightly-delayed replication setup like most databases provide, or a double-write system. These aren't backups because they make it impossible to recover from human errors, which include obvious things like dropping the wrong table, but also less obvious things, like a subtle bug that corrupts/damages some records and may take days or weeks to notice. Your standbys/mirrors are going to copy both of obvious and non-obvious things before you have a chance to stop them.
This is one of the most important things to remember. Redundancy is not backup. Redundancy is redundancy and it primarily protects against hardware and network failures. It's not a backup because it doesn't protect against human or software error.
3. regularly verified by real-world restoration cases; backups can't be trusted until they're confirmed, at least on a recurring, periodic basis. Automated alarms and monitoring should be used to validate that the backup file is present and that it is within a reasonable size variance between human-supervised verifications. Automatic logical checksums like those suggested by some other users in this thread (e.g., run pg_restore on a pg_dump to make sure that the file can be read through) are great too and should be used whenever available.
4. complete, consistent, and self-contained archive up to the timestamp of the backup. Differenced backups count as long as the full chain needed for a restoration is present.
This excludes COW filesystem snapshots, etc., because they're generally dependent on many internal objects dispersed throughout the filesystem; if your FS gets corrupted, it's very likely that some of the data referenced by your snapshots will be corrupted too (snapshots are only possible because COW semantics mean that the data does not have to be copied, just flagged as in use in multiple locations). If you can export the COW FS snapshot as a whole, self-contained unit that can live separately and produce a full and valid restoration of the filesystem, then that exported thing may be a backup, but the internal filesystem-local snapshot isn't (see also point 1).