My team and I switched from bitbucket to gitlab a few months ago and we love the transition. Gitlab provides a lot of value to me and my team who are learning how to code while working on side projects. Although we cannot send merge requests today because of this issue, we are all cheering them on. I’m very happy that they are so transparent about their issues because my team and I are learning so much from their rep…
GitLab Database Incident – Live Report
251–260 of 621 posts
Re: GitLab Database Incident – Live Report
#252Earlier quoted context omitted.
5 interviews isn't -that- insane, it's on the high end though. But I do know for sure they have turned down extremely technically gifted people for "non-technical" reasons, the kind of people that would ensure this sort of disaster wouldn't happen. So maybe the process really isn't that great.
I wondered why they always have so many job reqs out, and why they're out for so long. It always seemed like an interesting organization to me, and I like the product, but everything about their hiring process seems to be shoeing people off. Perhaps this will be incentive to try onboarding people or at least to bring in some consultants to shore them up.
This is an issue during hiring when companies have minimal Ops background. They (usually) attempt to interview the candidate as they would a normal dev position. This has 2 serious problems. 1) Real Ops candidates are going to seem lacklustre compared to web devs when giving web dev coding tasks and inteview questions. 2) You probably aren't going to ask them anything actually relevant to their ability to do Ops well because you just don't know.
This can usually be solved by hiring an experienced SRE manager. However those people are expensive and hard to find.
The next best thing is to have a gun SRE in your personal network that you can convince to join your team and help you build it out.
If you can't achieve that for whatever reason then you really just have to hire based on experience and suck up the cost of hiring an ex-Google/Facebook/Netflix/Twitter SRE that you can be reasonably confident will be good.
That is still hard though because those sorts of people know there is a ton of shit shoveling to be done if you are the first real SRE on board.. you have to clean up the mess left in the wake of "devs doing ops" which is not Devops btw. Devops is what is done by SREs. "devs doing ops" is how this sort of thing happens.
/rant
Re: GitLab Database Incident – Live Report
#253We're using the Gitlab CE for our internal use, but one of our customer's is on Gitlab.com. We're likely still going to recommend Gitlab.com after this, but also enable mirroring to our internal instance. As a side note, I just checked our S3 Gitlab backup bucket and it does have backups for every day for the last year (1.8 GB each yikes) so instead of failing to create the backups, its actually failing to delete the…
Re: GitLab Database Incident – Live Report
#254Earlier quoted context omitted.
What if I told you all of society is held together by duct tape? If you're surprised that startups cut corners you're in for a rude awakening. I'm frequently amazed anything works at all.
The real question is what holds together duct tape?
Re: GitLab Database Incident – Live Report
#255As I read the report I notice a lot of PostgreSQL "backup" systems depend on snapshotting from the FS & Rsync. This may work for database write logs, but it certainly will corrupt live git repositories that use local file system locking guarantees. NFS also requires special attention (a symlink lock) as writes can be acknowledged concurrently for byte offsets unless NFSv4 locking & compatible storage software is used…
The link discusses why rsync and tarballs are not good backup solutions. But, those wouldn't rightly be called "snapshots". "Snapshot" implies atomic, right? Surely an atomic snapshot would not corrupt git -- I would expect git, like any database, is designed to be recoverable after power failure, to which recovering from an atomic snapshot should be equivalent.
Or is that not the case?
Re: GitLab Database Incident – Live Report
#256BTW, overhosting is always a big risk, since it takes along period to catch up for the incident.
Re: GitLab Database Incident – Live Report
#257I'm a huge Gitlab fan. But I long ago lost faith in their ability to run a production service at scale. Nothing important of mine is allowed to live exclusively on Gitlab.com. It seems like they are just growing too fast for their level of investment in their production environment. One of the only reasons I was comfortable using Gitlab.com in the first place was because I knew I could migrate off it without too much…
Like you, I would like to add my 2 cent, which I hope will be taken positively, as I would like to see them provide healthy competition for GitHub for years to come. Since GitLab is so transparent about everything, from their marketing/sales/feature proposals/technical issues/etc., they make it glaringly obvious, from time to time, that they lack very fundamental core skills, to do things right/well. In my opinion, t…
For those who haven't been following along, Gitlab's compensation policy is pretty much intentionally designed to not pay people to live in SF. It's a somewhat reasonable strategy for an all remote company. But they seem to have some pretty ambitious plans that may not be compatible with operating a physical plant.
Re: GitLab Database Incident – Live Report
#258Earlier quoted context omitted.
Hey Daniel, I want to thank you for your candid feedback. Rest assured that this sort of thing makes it back to the team and is truly appreciated no matter how harsh it is. You're absolutely right -- we need to do better. We're aware of several issues related to the .com service, mostly focused on reliability and speed, and have prioritized these issues this quarter. The site is down so I can't link directly, but her…
I'm a bit curious here. Do you think that your issues with scalability and reliability have to do with your tech choice (I think it was Ruby on Rails)? Don't want to bash Rails, I'm just genuinely curious, since I come from a Rails background as well and have seen issues similar to yours in the past.
Just looking at their gemfile is rather telling: a couple hundred gems. I've always felt that if you're going above 100, you should carefully consider how much your codebase is trying to achieve.
They're probably at the point where they really want to think about splitting off of their monolith codebase and into microservices.
Re: GitLab Database Incident – Live Report
#259I noticed the issue when I was pushing code earlier today. Hopefully this gets resolved soon. You guys are doing a great job. Keep up the good work!
Thanks, not feeling great about the job we're doing today, but we'll learn from this.
Re: GitLab Database Incident – Live Report
#260Earlier quoted context omitted.
On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.
And then your disk fills because your new hire doesn't know about trash and you have an outage and remove it