541–550 of 621 posts
I wonder if this could have been avoided by using different subdomains. For example, instead of db1.cluster, what if it were named db-production.cluster. Would it have still happened? Probably not. I could totally see myself accidentally typing db1 out of muscle memory, but there's no way to accidentally type db-production.
That's what auto-complete is for.
rm -rf dbEarlier quoted context omitted.
> You shouldn't let yourself get to that point. Your alerting system should alert you when disk is at 70% or something with a ton of margin. What would your recommendation be?
Monitor the rate at which the disk is filling up, and extrapolate that to when it will hit 90%. If that time is outside business hours, alert early. If current time is not in business hours, alert later if possible.
I'm a huge Gitlab fan. But I long ago lost faith in their ability to run a production service at scale. Nothing important of mine is allowed to live exclusively on Gitlab.com. It seems like they are just growing too fast for their level of investment in their production environment. One of the only reasons I was comfortable using Gitlab.com in the first place was because I knew I could migrate off it without too much…
Hey Daniel, I want to thank you for your candid feedback. Rest assured that this sort of thing makes it back to the team and is truly appreciated no matter how harsh it is. You're absolutely right -- we need to do better. We're aware of several issues related to the .com service, mostly focused on reliability and speed, and have prioritized these issues this quarter. The site is down so I can't link directly, but her…
A sales rep reached out and I told him we would be happy to pay if that's required to be able to use the cloud hosted version reliably but I got no response. Certainly we could host GitLab EE or CE on our own but this is what we wanted to avoid and leave it to those who know it best. xp-dev never ever had any downtime longer than 10 minutes that we actively used during the last 6 years. I'm still paying them so that I can search older projects as the response time is instant while gitlab takes more than 10 seconds to search.
Besides the slow response times and frequent 500 and timeout errors that we got accustomed to, gitlab.com displays the notorious "Deploy in progress" message every other day for over 20-30 minutes preventing us from working. I really hoped that 6-7 months would be enough time to sort these problems out but it only seems to be worsening and this incident kinda makes it more apparent that there are more serious architectural issues, i.e. the whole thing running on one single postgresql instance that can't be restored in one day.
We have one gitlab issue on gitlab.com to create automated backups of all our projects so that we could migrate to our own hosted instance (or perhaps github) but afair gitlab.com does not support exporting the issues. This currently locks us into gitlab.com.
On one hand I'm grateful to you guys because of the great service as we haven't paid a penny, on the other hand I feel that it was a big mistake picking gitlab.com since we could be paying GitHub and be productive instead of watching your twitter feed for a day waiting for the postresql database to be restored. If anyone can offer a paid hosted gitlab service that we could escape to, I'd be curious to hear about.
This is why you have lots of copies of your data. Matt Raney has a great talk about designing for failure that includes details on Uber's "worst outage ever." It too involved postgresql replication and mistaking on host for another, but they didn't lose any data because they had more than a dozen live copies of their database. https://www.youtube.com/watch?v=bNeZYVIfskc&t=26m54s This isn't an alternative to working b…
Earlier quoted context omitted.
That guy is going to be interviewing at some company with someone who's obsessive enough about outage reports to remember a then-obscure one years later, but enough of an idiot to not understand that people aren't personally to blame for this sort of stuff? Sounds like even in that very contrived scenario the guy involved would dodge a bullet in not being hired by a bunch of idiots.
"I worked at GitLab." Googles name + GitLab, finds postmortem Highly likely, and now you don't get to tell your own story and emphasize what you want to.
What's your most valuable lesson from that incident?
You're hired!
Earlier quoted context omitted.
You can check out their codebase here: https://github.com/gitlabhq/gitlabhq Just looking at their gemfile is rather telling: a couple hundred gems. I've always felt that if you're going above 100, you should carefully consider how much your codebase is trying to achieve. They're probably at the point where they really want to think about splitting off of their monolith codebase and into microservices.
Maybe it's because I'm familiar with almost all of the gems, but I don't see anything wrong with their Gemfile. It's a pretty complex project, and they really do have a ton of integrations and features that need those gems. There's probably a few small libraries that they could have rewritten in a few files (never a few lines), but what's the point? The version is locked, and code can always be forked if they need to…
You'd be surprised what you can do by carefully considering what the desired outcome actually needs to be.
Maybe there is justification for all the gems in gitlab's Gemfile, I didn't go through it with a fine tooth comb - but this reaffirms my experience that complex projects outgrow monolith codebases. Having an infrastructure outage take down your entire business is kind of a symptom of that.
As I read the report I notice a lot of PostgreSQL "backup" systems depend on snapshotting from the FS & Rsync. This may work for database write logs, but it certainly will corrupt live git repositories that use local file system locking guarantees. NFS also requires special attention (a symlink lock) as writes can be acknowledged concurrently for byte offsets unless NFSv4 locking & compatible storage software is used…
I am the sole back end developer for a greenfield web application that is very data heavy. The application is still in alpha at the moment, but part of the development process involved prepopulating the database with about 10 mil rows or so spread out over about 15 tables. Nothing too crazy. However, once the application is launched I expect to have exponential growth in data due to the nature of the application.
Currently, this application is set up on Linode. The database server is standalone and I have the ability to spin up multiple application machines and a load balancer. Each of these application machines read from and write to the single database machine. The database machine itself has full disk image backups taken every 24 hours, 7 days, and month. I also do a manual snapshot from time to time. On top of this I also usually dump into a tar file on my own external drive every once and a while as well. I'm fairly new to devops stuff and most of my experience involves building applications and not necessarily deploying them. I'm wondering if what I'm doing as far as backups and stability is enough or if I should be incorporating other methods as well. Have any thoughts?
They say that git data (repos and wikis) are not affected... well ... if they would have had their PRs and Issues in git repositories, too... Disclaimer: Worked on a POC for exactly this last semester and going to publish my results in the next few weeks.
Earlier quoted context omitted.
I myself initially added my name to the document in various parts, this was later changed to just initials. I specifically told my colleagues it was OK to keep it in the document. I have no problems taking responsibility for mistakes, and making sure they don't happen ever again.
That's awesome, but why publicize it? This isn't an act of contrition for you, no one outside your team really needs to see your dirty laundry, and actually comes off as unprofessional to me. The gitlab team is a team, and you take responsibility as a team. Placing names and initials in the liveblog makes it look like SOMEONE is trying to assign and pass off blame, even if that is not what is happening. Presumably in…
Earlier quoted context omitted.
Anybody whose opinion matters understands that this type of event is a process problem, not a person problem. GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names). To be honest, through all of the comments, yours seems the mo…
In a few years the guy doing the `rm -rf` is going to be on a job interview and someone will recall bits of this report. Enough bits to remember the guy, not enough bits to remember that it wasn't his (individual) fault. Transparency doesn't mean publicly throwing people under the bus. I'm not a GitLab customer, I'm relaxed. :)
One of the SAN arrays didn't come up, and then started rebuilding itself. Our storage was one of those multi-million dollar contracts from IBM. They flew a guy out to the University and after a lot of work, they said the array was lost and unrecoverable.
Backups for production for some VMs were on virtual tape .. on the same shelves as production. O_o
At least a lot of our clusters were split between racks, so in many cases we could just clone another one. We learned that MS BizSpark, in a cluster, only puts the private key on half the machines. We had to recreate a bunch of BizSpark jobs based off what we could still see in the database and our old notes and password vaults. We had been planning on upgrading to a newer version of BizSpark on a Server 2012 (it was on 2003), so this kinda forced us to. Shortly afterwards we learned how to make powershell scripts to backup those jobs and test the backups by redeploying them to lower environments.
The sys admin over the backups was looking for a new job. You can't really fire people from universities easily, because it's very difficult to find IT staff that will take university wages. Word was out though, if he didn't find new work, he was going to be let go. Not laid off, made redundant, or have his position removed. He would be fired.