141–150 of 621 posts
This has reminded me how important it is to perform regular rehearsals of data recovery scenarios. I'd much rather find these failures in a practice run. Thanks to GitLab for continuing to openly share their experience.
This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…
We have 2 people on rotation and anything like this you pair with someone else and talk through it. As we like to say "It's only a mistake if we both make it."
The military does a very similar thing. An Army company commander usually has a RTO (radiotelephone operator) to handle taking on the radio. This frees the commander to make real-time decisions and response quickly to the situation on the ground, and frees him/her from having to spend time explaining things. A really good RTO will function as the voice of the commander, anticipating what he/she needs to get from the person on the other end of the radio. This is a great characteristic of a good operations engineer, too. While the person doing triage is addressing the problem, the communicator is roping in other resources that might be needed and communicating the current situation out to the rest of the team/company
> Our backups to S3 apparently don’t work either: the bucket is empty followed by > So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. is no way to be running a public service with customer data. Did the person who set up that S3 job simply write a script or something and just go "yep, it's done" and walk away? Seriously?
Apparently the following insane interviewing process, wasn't enough to find someone competent enough to cover the basics. https://about.gitlab.com/jobs/production-engineer/ ------------------- Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be con…
But I do know for sure they have turned down extremely technically gifted people for "non-technical" reasons, the kind of people that would ensure this sort of disaster wouldn't happen.
So maybe the process really isn't that great.
Earlier quoted context omitted.
I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.
I've done this exact thing my my servers. Also like GL I prepend PRODUCTION and STAGING on PS1. I should probably make the PRODUCTION flash just in case.
> Our backups to S3 apparently don’t work either: the bucket is empty followed by > So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. is no way to be running a public service with customer data. Did the person who set up that S3 job simply write a script or something and just go "yep, it's done" and walk away? Seriously?
Apparently the following insane interviewing process, wasn't enough to find someone competent enough to cover the basics. https://about.gitlab.com/jobs/production-engineer/ ------------------- Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be con…
The reason for my rejection? I had an ongoing side project and development/consulting gig which had been paying the bills for a few years which they thought I 'misrepresented'. I guess they assumed I was still working there (I was and would continue to do consulting and development on my own time), but the biggest LOL of the whole shitty process was their reasoning: "We are very risk-averse in our hiring." No shit.
I guess they weren't risk-averse enough in their operations though, so I'm glad I didn't get through. It sounds like it would have been an uphill battle all the way to make changes to keep things sane.
And having to crowdsource your move from cloud to your own hardware? Listen to your people, they already had good ideas, all of which were documented in your open issue tracker.
Also helpful to make the window background color different, or some other highly conspicuous visual difference when working on several very similar production machines.
Earlier quoted context omitted.
It's both. I very much appreciate their forthrightness and the way they conduct their company generally. Having said that, I have the code I work on, related content, and a number of clients on the service. [edit for additional point] They need the infrastructure guy they've been looking for sooner than later. I hope there's good progress on that front.
We've hired some great new people recently but as you can see there is still a lot work to do. https://about.gitlab.com/jobs/production-engineer/
Earlier quoted context omitted.
Also, as a safety net, sometimes you don't need to run `rm -rf` (a command which should always be prefaced with 5 minutes of contemplation on a production system). In this case, `rmdir` would have been much safer, as it errors on non-empty directories.
These days, I've been very implicit in how I run rm. To the extent that I don't do rm -rf or rmdir (edit: immediately), but in separate lines as something like: pushd dir ; find . -type f -ls | less ; find . -type f -exec rm '{}' \; ; popd ; rm -rf dir It takes a lot longer to do, but I've seen and made enough mistakes over the years that the forced extra time spent feels necessary. It's worked pretty well so far --…
find ... -delete