371–380 of 621 posts
Earlier quoted context omitted.
Thanks, not feeling great about the job we're doing today, but we'll learn from this.
The way you are open about this and do not blame the engineer who did "rm -Rvf" (he knows he fucked up, suffers enough already), and seeing that improvements can be made and you're willing to do them. Applying for work with you now, and moving all my stuff to GitLab.
For example, instead of db1.cluster, what if it were named db-production.cluster. Would it have still happened? Probably not.
I could totally see myself accidentally typing db1 out of muscle memory, but there's no way to accidentally type db-production.
Earlier quoted context omitted.
Two points: 1) Patio11 touches on a very good lesson, in passing, in an article about Japanese business[1]: While raw programming ability might not be highly valued at many Japanese companies, and engineers are often not in positions of authority, there is nonetheless a commitment to excellence in the practice of engineering. I am an enormously better engineer for having had three years to learn under the more senior…
One of thing that I learned the hard way about "Japanese companies" - despite western conceptions, every company in Japan has its own unique culture (and takes pride in having its own culture!). What's more, often departments and division inside the same companies work in very different way. Why am I saying that? Because in some of the Japanese companies I've worked with were the exact opposite of that. To be sure, l…
My opinion: script it. Always. It doesn't matter if it's ansible, bash, puppet, python, whatever, just make sure it's not an ad-hoc command. Test the script on a server which can be sacrificed. Test as long as there is a single glitch. Run it in production.
It's to eliminate typos and to have a "log" to see what actually had been done.
Earlier quoted context omitted.
Also, as a safety net, sometimes you don't need to run `rm -rf` (a command which should always be prefaced with 5 minutes of contemplation on a production system). In this case, `rmdir` would have been much safer, as it errors on non-empty directories.
I am actually curious what user they were logged in as and what permissions were in effect. Unfortunately, the answer most places is that the diagnostic account (as opposed to the corrective action account) is fully privileged (or worse, root).
Earlier quoted context omitted.
This sounds good until you consider that many systems are utilizing multiple drives. When someone is expecting to delete a large file and it ends up on a different drive, problems could arise.
Renaming a file should not move its data, right? So rename the file into .file.$(date -Iseconds).trash (but make sure that no legitimate files are ever named in this pattern). Then put that file path into a global /var/trashlist. To cleanup, you just check that file for expired trash and make the final deletion.
"Of around 310 GB only about 4.5 GB is left" What is gitlab storing in their database? From what I understand, the repos were untouched by the DB problems, so what is taking up a third of a terabyte of DB space?
Earlier quoted context omitted.
On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.
And then your disk fills because your new hire doesn't know about trash and you have an outage and remove it
This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…
On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.
Earlier quoted context omitted.
Maybe https://www.youtube.com/watch?v=8W4cfIyNvts ?
I didn't hear any "I agree" in that one. I found this one which does: https://www.youtube.com/watch?v=PAaMuTRGP6k&t=1h7m30s
Earlier quoted context omitted.
What if I told you all of society is held together by duct tape? If you're surprised that startups cut corners you're in for a rude awakening. I'm frequently amazed anything works at all.
The real question is what holds together duct tape?