Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix Admin Horror Story Summary (1992)

www-uxsup.csx.cam.ac.uk

Post reply on HN
Print view View original

51–60 of 94 posts

Re: Unix Admin Horror Story Summary (1992)

#51

Post by bityard »

hulitu wrote: post #36
rntksi wrote: post #2

>Well one time I was installing a minimal base system of Linux on a friends PC, so that we would have all the necessary utlitities to bring over the rest of the stuff. His 3 1/2 inch disk was dead, so when had to get the 5 1/4 inch version of the boot/root disk. Too bad that version, having to fit in 1.2M instead of 1.44, didn't have tar Heh ... I wonder how many years forward will people stop knowing what a 3 1/2 an…

Ah, the good old days. Single density, double density ( 720 kB, 1,44 MB). I heard also of 2.88 MB floppies - never saw one in real life. If i remember correctly 2.88 MB was double density double sided and you needed a special floppy drive.

Single-sided: 360 KB, double-sided (or double density): 720 KB, high density: 1.44MB.

There were 2.88 MB disks and drives but they never gained much traction, because they were expensive and the PC industry kept promising various "floppy killers" like the ZIP drive.

Re: Unix Admin Horror Story Summary (1992)

#52

Post by greedo »

I had a contractor installing software last week. He was granted full sudo permissions despite having demonstrated a unique set of command line skills. Everything was going fine until I get a text from his manager saying the contractor couldn't SSH in this morning. Turns out he had gotten frustrated with some file/folder permissions in the directory where he was supposed to install the software package. So he simply ran `chmod -R 777 /*`

Needless to say that required a full restore from the previous night's backup since he hadn't snapshotted the VM before beginning his work. He was very angry that he had lost 2 days of work. I was very sympathetic...

Re: Unix Admin Horror Story Summary (1992)

#53

Post by floren »

It really drives home how expensive storage used to be when you see how many stories are variations on, "In an attempt to free up some disk space, I deleted a critical system file".

The version for those of us who came of age in the 90s is wondering what all this crap in C:\Windows\System32 is, and whether deleting it will give us enough space to install Age of Empires.

Re: Unix Admin Horror Story Summary (1992)

#54

Post by dsr_ »

treesknees wrote: post #29
whartung wrote: post #10

Worst thing I ever did was cross hard mount NFS volumes across two machines. With a hard NFS mount, the mount will hang until the other machine responds. When we had to power cycle the two servers, they would not come up as they were deadlocked waiting for each other. That was exciting.

This is somewhat common in environments with stable power, you basically never have the entire IT system go down and come back up at the same time.

"basically never" is guaranteed to happen at a particularly terrible time, and you won't have recovery procedures, because why would you bother?

Re: Unix Admin Horror Story Summary (1992)

#55

Post by dsr_ »

hulitu wrote: post #36
rntksi wrote: post #2

>Well one time I was installing a minimal base system of Linux on a friends PC, so that we would have all the necessary utlitities to bring over the rest of the stuff. His 3 1/2 inch disk was dead, so when had to get the 5 1/4 inch version of the boot/root disk. Too bad that version, having to fit in 1.2M instead of 1.44, didn't have tar Heh ... I wonder how many years forward will people stop knowing what a 3 1/2 an…

Ah, the good old days. Single density, double density ( 720 kB, 1,44 MB). I heard also of 2.88 MB floppies - never saw one in real life. If i remember correctly 2.88 MB was double density double sided and you needed a special floppy drive.

A quirk of accessibility of low-level formatting meant that you could persuade a 1.44MB disk drive to create a 1.72MB disk reliably, and a 1.76MB disk slightly less reliably.

IIRC, Microsoft decided to use the 1.72MB quirk for physical distribution of their software, since it made a significant cost savings in those days before cheap optical media, in those days before network distribution of commercial consunmer software.

Re: Unix Admin Horror Story Summary (1992)

#56

Post by yjftsjthsd-h »

geocrasher wrote: post #48

This one made me LOL: My mistake on SunOS (with OpenWindows) was to try and clean up all the '.*' directories in /tmp. Obviously "rm -rf /tmp/*" missed these, so I was very careful and made sure I was in /tmp and then executed "rm -rf ./.*". I will never do this again. If I am in any doubt as to how a wildcard will expand I will echo it first. I read this, and just had to go try it because I couldn't picture it in my…

Huh, what do you know. I thought modern rm only protected against deleting / by absolute path, but it looks like it'll protect you from deleting your parent regardless:

    $ docker run --rm -ti debian:11  # sandbox the danger...
    root@c70dde9f38a3:/# cd /tmp
    root@c70dde9f38a3:/tmp# rm -rf ./.
    rm: refusing to remove '.' or '..' directory: skipping './.'
    root@c70dde9f38a3:/tmp# mkdir -p /tmp/1/2/3/4/5
    root@c70dde9f38a3:/tmp# cd /tmp/1/2/3/4/5
    root@c70dde9f38a3:/tmp/1/2/3/4/5# rm -rf ./.
    rm: refusing to remove '.' or '..' directory: skipping './.'

Re: Unix Admin Horror Story Summary (1992)

#57

Post by aidenn0 »

bityard wrote: post #51
hulitu wrote: post #36

Earlier quoted context omitted.

Ah, the good old days. Single density, double density ( 720 kB, 1,44 MB). I heard also of 2.88 MB floppies - never saw one in real life. If i remember correctly 2.88 MB was double density double sided and you needed a special floppy drive.

Single-sided: 360 KB, double-sided (or double density): 720 KB, high density: 1.44MB. There were 2.88 MB disks and drives but they never gained much traction, because they were expensive and the PC industry kept promising various "floppy killers" like the ZIP drive.

The superdisk were what I wish had caught on. They could read HD floppies (though not any 2.88MB formats) and at some iteration were able to treat a single HD floppy as a 32MB(!!) tape drive.

IOMega seemed to have better marketing though.

Re: Unix Admin Horror Story Summary (1992)

#58

Post by medstrom »

hnlmorg wrote: post #43
geocrasher wrote: post #23

2003 or 2004. Customer called in and said that his dedicated server was hacked. I restored from backup. An hour later, he calls back. Hacked again . Restored again. An hour later, he calls back. He realizes that the hacker is him ! He's doing a thing, but doesn't know what he's doing wrong. So I have him email me the last thing he typed on his server, as root: rm -rf /home/user/path/to/thing /home/otheruser/path/to/s…

This is precisely why I always `-v` when `rm`ing recursively. It might be closing the barn door after the proverbial horse has bolted; but at least the fuck up is visible and in some circumstances you have a fighting chance to kill `rm` before too much damage has been done.

Is there a rm variant that bundles an undo command?

Re: Unix Admin Horror Story Summary (1992)

#59

Post by gorgoiler »

Alas, as cam.ac.uk prepares for the final shutdown of Hermes — their quarter century old Unix mail facility used by members great and small, from telnet to prayer and replaced by Microsoft Outlook on 12/31 — the timing of this post could not be more prescient.

Another onprem service formerly run by experts shutters its doors. In their halcyon days they wrote Exim.

You can’t even forward to gmail because the current sysadmin team have IPv6 set up without PTRs, causing gmail to 500 one in every N deliveries.

I imagines GOOG and the University of Cambridge locked in an endless battle of “don’t you know who I am”, and meanwhile the dons et al shuffle into the abyss of outlook.com penury.

What would ‘fanf2 do?

Re: Unix Admin Horror Story Summary (1992)

#60

Post by hnlmorg »

medstrom wrote: post #58
hnlmorg wrote: post #43

Earlier quoted context omitted.

This is precisely why I always `-v` when `rm`ing recursively. It might be closing the barn door after the proverbial horse has bolted; but at least the fuck up is visible and in some circumstances you have a fighting chance to kill `rm` before too much damage has been done.

Is there a rm variant that bundles an undo command?

There's a few ways to solve this:

+ a command that doesn't actually delete but rather moves the file to a trash folder that gets emptied periodically (this is exactly how a desktop environments work)

+ a command that removes the inode but leaves the data intact. This could get pretty messy pretty quickly on SSDs.

+ `rm` behaves as normal but you restore the file using forensic tools / file recovery tools. Similar problem with SSDs as above.

+ `rm` behaves as normal but you run a snapshotting file system such as ZFS (maybe even have a wrapper function around `rm` to take a snapshot before running `/bin/rm`). Any "undo" would be recovering from snapshot.

+ `rm` behaves as normal but you restore from backup. Which isn't really a solution but still worth presenting here given the range of options I've presented.

Personally I went with ZFS where possible.

If recovery is tricky and I'm throwing wildcards at the problem then I'll do an `ls` first just to double check I'm not doing anything stupid.

Post reply on HN