"Our backups to S3 apparently don’t work either: the bucket is empty" 6/6 failed backup procedures. Looks like they are going to be hiring a new sysadmin/devops person...
Like TJ watson said "fire you? I just spent ten million dollars training you."
GitLab Database Incident – Live Report
111–120 of 621 posts
Re: GitLab Database Incident – Live Report
#112For item 3h under recovery, consider: chattr +i /var/opt/gitlab/postgresql/data Yes, it doesn't completely stop foot-guns, but it means you have to shoot twice [0]. [0]: chattr -i /whatever rm /whatever
Does that prevent postgres from modifying it?
Re: GitLab Database Incident – Live Report
#113Earlier quoted context omitted.
It's both. I very much appreciate their forthrightness and the way they conduct their company generally. Having said that, I have the code I work on, related content, and a number of clients on the service. [edit for additional point] They need the infrastructure guy they've been looking for sooner than later. I hope there's good progress on that front.
We've hired some great new people recently but as you can see there is still a lot work to do. https://about.gitlab.com/jobs/production-engineer/
There's also this one if anyone is interested in helping out with the DB side of things: https://about.gitlab.com/jobs/specialist/database/
Re: GitLab Database Incident – Live Report
#114Earlier quoted context omitted.
I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.
Nope, that would be me.
I've been there myself, it was at the start of my career and almost ended it. I know how incredibly emotional this kind of thing can be. Just understand you aren't the first, you wont be the last and shit happens. If you are ever in Philly I'll buy you a beer if you drink, and a dinner if you don't.
Re: GitLab Database Incident – Live Report
#115Earlier quoted context omitted.
I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.
All of my non-production machines have emojis in PS1 somewhere. It sounds ridiculous, but I know that if I see a cheeseburger or a burrito I'm not about to completely mess everything up. Silly terminal = silly data that I can obliterate.
I think I'd rather make the production systems stand out, and add the emojis there. My prompts have a red background, but emoji prompts just tickle me, somehow.
Re: GitLab Database Incident – Live Report
#116This is why you have lots of copies of your data. Matt Raney has a great talk about designing for failure that includes details on Uber's "worst outage ever." It too involved postgresql replication and mistaking on host for another, but they didn't lose any data because they had more than a dozen live copies of their database.
https://www.youtube.com/watch?v=bNeZYVIfskc&t=26m54s
This isn't an alternative to working backups, of course, but it is an additional safety net. Plus it can give you a lot more options when handling an incident.
Re: GitLab Database Incident – Live Report
#117I noticed the issue when I was pushing code earlier today. Hopefully this gets resolved soon. You guys are doing a great job. Keep up the good work!
Thanks, not feeling great about the job we're doing today, but we'll learn from this.
More than loving your service we like you guys as people and trust you to get things straightened out. I wouldn't host my code with anyone else despite this.
Re: GitLab Database Incident – Live Report
#118> 2017/01/31 23:00-ish
> YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory.
> After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com
> 2017/01/31 23:27
> YP terminates the removal, but it’s too late. Of around 310 GB only about 4.5 GB is left
The naming couldn't be more confusing
Re: GitLab Database Incident – Live Report
#119Earlier quoted context omitted.
I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.
Nope, that would be me.
Yorick! Thank you for the transparency, I know how tough incidents like these can be. Stop the bleeding, figure out how to handle this better in the future, but most of all, take care!
Re: GitLab Database Incident – Live Report
#120>1. LVM snapshots are by default only taken once every 24 hours. YP happened to run one manually about 6 hours prior to the outage >2. Regular backups seem to also only be taken once per 24 hours, though YP has not yet been able to figure out where they are stored. According to JN these don’t appear to be working, producing files only a few bytes in size. >3. SH: It looks like pg_dump may be failing because PostgreSQ…
What if I told you all of society is held together by duct tape? If you're surprised that startups cut corners you're in for a rude awakening. I'm frequently amazed anything works at all.