Earlier quoted context omitted.
This is usually true, except when it's not : I have personally experienced a near-catastrophic situation 3 years ago, where 13 out of 15 days' worth of nightly RDS MySQL snapshots were corrupt and would not restore properly. The root cause was a silent EBS data corruption bug (RDS is EBS-based), that Amazon support eventually admitted to us had slipped through and affected a "small" number of customers. Unlucky us. W…
Snapshots are not backups, although many people use them as backups and believe they are good backups. Snapshots are snapshots. Only backups are backups.
Postmortem of database outage of January 31
221–230 of 269 posts
Re: Postmortem of database outage of January 31
#222Great to have a full-featured, professional post-mortem. Incidentally I work at a company that suffered data loss because of this outage and we're looking for ways to move out of GL. My 2 cents... I might be the only one, but I don't like the way GL handled this case. I understand transparency as a core value and all, but they've gotten a bit too far. IMHO this level of exposure has far-reaching, privacy implications…
Hi Panagiotis. I'm glad to hear you like the postmortem. I'm very sorry your company suffered data loss. If you want to move from GitLab.com please know that you can easily export projects and import them on a self-hosted instance https://gitlab.com/help/user/project/settings/import_export.... (and if in the future we regain your trust you can also go the other way). It is not our intent to have one of our team membe…
> It is not our intent to have one of our team members implicated by the transparency. That is why we redacted their name to team-member-1 and in any future incidents we'll do the same.
Great, good to know. I wish all the success in the world to you and everyone involved with Gitlab.
Re: Postmortem of database outage of January 31
#223Does anyone have a link to the YouTube stream they's talking about? Can't seem to find it on their channel. And the link in the doc is redirecting to the live link [1] which doesn't list the stream. [1] - https://www.youtube.com/c/Gitlab/live
Thanks for taking the interest to check it out.
It's an unlisted YT video, so that's why it might be hard to find.
Here it is: https://www.youtube.com/watch?v=nc0hPGerSd4
Re: Postmortem of database outage of January 31
#224Earlier quoted context omitted.
Hi Panagiotis. I'm glad to hear you like the postmortem. I'm very sorry your company suffered data loss. If you want to move from GitLab.com please know that you can easily export projects and import them on a self-hosted instance https://gitlab.com/help/user/project/settings/import_export.... (and if in the future we regain your trust you can also go the other way). It is not our intent to have one of our team membe…
Thanks for the reply. > It is not our intent to have one of our team members implicated by the transparency. That is why we redacted their name to team-member-1 and in any future incidents we'll do the same. Great, good to know. I wish all the success in the world to you and everyone involved with Gitlab.
Re: Postmortem of database outage of January 31
#225Earlier quoted context omitted.
RDS has severe performance limitations as in you can't provision more than 30K IOPS which is about 1/2 the performance of low end consumer SSD and about 1/20 the performance of a decent PCI-E SSD. You way better of running on decent dedicated hardware for the DB.
You can get 500K random reads per second and 100K random writes per second using RDS Aurora. If you truly need more than 30K IOPs, I would recommend leveraging read-replicas, a Redis cache, and other solutions before just "throwing money at the problem" and purchasing a million IOPs.
Re: Postmortem of database outage of January 31
#226My main question is still: >> Why did replication stop? - A spike in database load caused the database replication process to stop. This was due to the primary removing WAL segments before the secondary could replicate them. Is this a bug/defect in PostgreSQL then? Incorrect PostgreSQL configuration? Insufficient hardware? What was the root cause of Postgres primary removing the WAL segments?
The cause is bad configuration. PgSQL, Mongo, and MySQL all use a transaction stream like this for replication and they all have to put some kind of cap on it or risk running out of disk space, but the cap should be made sufficiently large to allow automatic resumption of disconnected slaves without manual redumping, except in extraordinary circumstances. Log retention should be long enough to last at least a long we…
innobackupex works fine locally on the server, streaming out to netcat or ssh on the remote side. Nothing wild like read only NFS required. It also copies all binlogs. Mydumper is pretty old at this point and doesn't do most of the things innobackupex can. I wouldn't recommend it.
Re: Postmortem of database outage of January 31
#227Earlier quoted context omitted.
>So I learned my first lesson with RDS - make sure final snapshot flag is enabled (for EC2 user, please remind yourself anything stored on ephemeral storage are going to be loss upon a hard VM stop/start operation, so backup!!!). This bit us once. Someone issued a `shutdown -h now` out of habit in an instance that was going for reboot, and it came back without its data, because "shutdown" is the same as "stop", and "…
Don't blame AWS because you don't understand what ephemeral storage is. There is a significant security reason why they blank the ephemeral storage. How would you feel if a competitor got the same physical server as you, and was able to read all of your data? AWS takes great lengths to protect customer data privacy in a shared, multi-tenant environment. They are very public through their documentation about how this…
AWS gets paid the big bucks to abstract such concerns away in a pleasant manner. The device with customer data can sit in reserve, attached to the customer's account, for a cooldown period (of maybe 24 hours?) that would allow the customer to redeem it. AWS could even charge a fee for such data redemptions to compensate for the temporary utilization of the resource, or they could say ephemeral instances will always cost your use + 1 day. They can put a quota on the number of times you can hop ephemeral nodes.
They could do basically anything else, because basically anything else is better than accidentally deleting data that you need due to a counterintuitive vendor-specific quirk that conflicts with established conventions and habits and then being told "Sorry, you should've read the docs better."
This is an Amazon-specific thing that bucks established convention and converts the otherwise-harmless habits of sysadmins into potential data loss events. It's very bad to do this ever (looking at you, killall Linux v killall Solaris), but it's especially bad to do it on a new platform like AWS where you know lots of people are going to be carrying over their established habits and learning the lay of the land. It is not reasonable for Amazon to tell the users that they just have to suck it up and read the docs more thoroughly next time.
This is not like invoking rm on your system or database root, which is a multi-decade danger that everyone is aware of and acclimated to accounting for, and which has multiple system-level safeguards in place to prevent it: user access control, safe-by-default versions of rm that have been distributed with most major distributions lately, etc., and for which thorough backup and replication solutions exist to provide remedies when inevitable accidents do happen.
The point is that just instantly deleting that data ASAP and providing 0 chance for recovery is wanton recklessness, and there's no excuse for it. Security is not an excuse because there's no reason they have to reallocate the storage the instant the node is stopped.
If such deletions could only be triggered from the EC2 console after removing a safeguard similar to Termination Protection, that may be more reasonable, but allowing a shutdown command from the CLI to destroy the data is patently irresponsible.
Good system design considers that humans will use the system, that humans make mistakes, and it will provide safeguards and forgiveness. Ephemeral storage fails on all of those fronts. Yes, technically, it's the user's fault for mistakenly pressing the buttons that make this happen. But that doesn't matter. The system needs to be reasonably safe. AWS's implementation of ephemeral storage is neither safe nor reasonable.
Amazon has done a good job of tucking ephemeral storage away. It used to be the default on certain instance sizes. As another commenter points out, it now requires one to specifically launch an AMI with instance-backed storage. It's good that they've made it harder to get into this mess, but it's bad that they continue to mistreat customers this way, especially when their prices are so exorbitant.
Re: Postmortem of database outage of January 31
#228Earlier quoted context omitted.
> Any half intelligent engineer would always first research good practices, pitfalls and existing information which has been gathered This is all nice theory, but in the end we're all humans. Pretending that you'll never make a serious mistake is just tat, pretending.
True, but no one is pretending to never make a mistake. There is a huge spectrum between never making a mistake and making the most fundamental mistake because developers were ignorant to 20 years of industry knowledge.
Re: Postmortem of database outage of January 31
#229Earlier quoted context omitted.
I had exactly the same feeling. One of my friends was a doctor at a hospital and there was a serious mistake which my friend reported to the consultant. The consultant made a good point that it was never just one mistake that caused the serious mistake; it was a series of smaller mistakes that hadn't been checked or addressed. (the argument was that there were processes to avoid these kinds of mistakes) If you read t…
I don't actually think it has anything to do with education. I think it really comes down to common sense. Any half intelligent engineer would always first research good practices, pitfalls and existing information which has been gathered from decades of other experienced engineers before doing anything stupid on their own. It seems that GitLab is lacking this attitude.
The decades of experience are not documented.
Re: Postmortem of database outage of January 31
#230The engineers still seem to have a physical server mindset rather than a cloud mindset. Deleting data is always extremely dangerous and there was no need for it in this situation. They should have spun up a new server to act as secondary the moment replication failed. This new server is the one you run all of these commands on, and if you make a mistake you spin up a new one. Only when the replication is back in good…
For me, personally, going from cloud servers to rented dedicated servers cut my bill by 93% – more than an order of magnitude. At same performance.
In fact, it’d be cheaper to run 10x as many dedicated servers than to use cloud solutions for me.