11–20 of 46 posts
You haven't lived until you unwittingly run rm -rf on a nfs mount of / on a remote box. Which happens to be the fileserver for a trading shop. In the middle of trading hours.
Yeah, been there, done that, although my first step is to umount and dd onto another volume so the sectors don't get reallocated and obliterate the data - and that's generally a sane first step for ANY recovery of this ilk. You haven't lived until you unwittingly run rm -rf on a nfs mount of / on a remote box. Which happens to be the fileserver for a trading shop. In the middle of trading hours.
I guess it's a good thing he didn't have an SSD and TRIM enabled!
SSD designers developed an interface allowing the operating system (e.g. Windows, Linux, Mac OS X etc.) to inform the controller that certain blocks are no longer in use via the TRIM command. This allows the internal garbage collector to electronically erase the content of these blocks, preparing them for future write operations.
http://forensic.belkasoft.com/en/why-ssd-destroy-court-evide...
Actually recovered quite a few people's homework assignments that way. Of course, you could also write to that file, so we had free-space sweepers. And since you could write to that empty space, you could also leave messages, even from the public (100,0) account, which had a zero permanent disk quota (all files deleted on logout).
A very important lesson regarding the Unix principles of "everything is a file" and programs that "do one job well".
A very important lesson regarding the Unix principles of "everything is a file" and programs that "do one job well".
Also a very important lesson regarding the dangers of poor user interface design. It's a little bit crazy that in 2014 we still have a significant amount of serious work being done on systems where a slip of the finger or a one-character typo in a script can literally destroy whole systems with no confirmation and no reliable recovery mechanism.
I once did recovery of this kind for a friend (using, I think, photorec or extundelete, not grep) and the hardest part by far was piecing together the "right" version of the files from all matching versions that were recovered from disk.
Yeah, been there, done that, although my first step is to umount and dd onto another volume so the sectors don't get reallocated and obliterate the data - and that's generally a sane first step for ANY recovery of this ilk. You haven't lived until you unwittingly run rm -rf on a nfs mount of / on a remote box. Which happens to be the fileserver for a trading shop. In the middle of trading hours.
Thankfully we had two machines which were virtually identical OS-wise (RedHat 6 if memory serves). I was able to get everything put back from the twin machine and keep everybody happy.
Thankfully that server kept running with relatively little issue the entire time even with all those core OS files gone. I don't think any customers were at all aware.
A very important lesson regarding the Unix principles of "everything is a file" and programs that "do one job well".
It's kind of flawed where he says " If you pick x to be big enough, you should get the entire file, plus a bit of junk around the edges." - because this is simply not how filesystems work - if you manage to get the full file, it's by luck that it was small enough for the filesystem to allocate its contents contiguously on the block device. Anything bigger and you're gonna quickly need something more sophisticated whi…
From a different perspective, hopefully /tmp is on a different filesystem from /home, otherwise the reading the man pages might overwrite the blocks you need to recover with the temporary files they produce. (And less, more, sort, etc.) Also, doing Google/StackOverflow searches is probably unwise due to the browser writing stuff to the 50 MB disk cache (FF default, anyway) on the filesystem you want. Probably step 1 should be "remount the partition read-only". Or better yet, "find another computer to use for research" :)
Earlier quoted context omitted.
Also a very important lesson regarding the dangers of poor user interface design. It's a little bit crazy that in 2014 we still have a significant amount of serious work being done on systems where a slip of the finger or a one-character typo in a script can literally destroy whole systems with no confirmation and no reliable recovery mechanism.
This is why every system I administer has 'rm' aliased to 'rm -i' (along with 'cp' and 'mv' just in case). I believe this is the default on RHEL/CentOS boxes. Certainly for root, but should be for every user. Sure, it can be a pain sometimes to have to confirm, but at least you get the chance....unless you add '-f'.
Glad I'm not the only one. :-)
However, that is rather a specific case, albeit a common one. I have lost count of how many times I've seen even very experienced sysadmins do something disastrous by accident that is entirely due to the poor usability of some Linux shell or other command line-driven software with a similar design style and culture.
I have seen someone nuke an entire system, with a shell script that failed at string interpolation and literally did an 'rm -rf /', after I explicitly warned them of the danger and they thought they'd guarded against it. That person was a very capable sysadmin with many years of experience, but expecting anyone to never make a mistake with that kind of system is like expecting a similarly experienced programmer to write bug-free code with nothing but an 80x25 terminal window and a line editor.