Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix Admin Horror Story Summary (1992)

www-uxsup.csx.cam.ac.uk

Post reply on HN
Print view View original

71–80 of 94 posts

Re: Unix Admin Horror Story Summary (1992)

#71

Post by mark-r »

drewg123 wrote: post #8
jlv2 wrote: post #6

I vaguely thought I posted in this thread back then. Back in 1984/5, I had a directory in my homedir called "etc" for of miscellaneous stuff. One day I thought: that's a bad name, I should remove it. I errantly typed "rm -rf /etc". Thankfully I got a "Permission Denied" error. Except, I then did the obvious override, "sudo rm -rf /etc" (1). This was on a VAX 11/780 with about 50 undergrads doing project work on. The…

When did it become ubiquitous? I first got root privs via sudo on a *nix system in 1991 or so, and I remember it being widely deployed even then.

When I first started using Unix in the late 1980's, there was no sudo. If I needed privileges I used su and got out of it as soon as I was able.

Re: Unix Admin Horror Story Summary (1992)

#72

Post by aleph- »

oneweekwonder wrote: post #34
patrickdavey wrote: post #18

Earlier quoted context omitted.

How did you fix it?

not op but we recently wanted to mount nfs and the sysadmin was adamant we use automounter[0] instead of fstab because if the nfs mount is not available it can hang the kernel. Not sure if it is true or just sysadmin lore but was interesting enough to learn about a alternative. [0]: https://linux.die.net/man/8/automount

Oh this can definitely happen, had it occur on my system till I started using nofail as a mount option.

Re: Unix Admin Horror Story Summary (1992)

#73

Post by tomlx »

geocrasher wrote: post #48

This one made me LOL: My mistake on SunOS (with OpenWindows) was to try and clean up all the '.*' directories in /tmp. Obviously "rm -rf /tmp/*" missed these, so I was very careful and made sure I was in /tmp and then executed "rm -rf ./.*". I will never do this again. If I am in any doubt as to how a wildcard will expand I will echo it first. I read this, and just had to go try it because I couldn't picture it in my…

Interestingly

  $ bash -c 'cd /var/empty; echo .*'
  . ..
but

  $ ksh -c 'cd /var/empty; echo .*'
  .*
Seems some ksh dev got bitten and restricted this ...

Re: Unix Admin Horror Story Summary (1992)

#74

Post by kottapar »

This was around 2009. A colleague in the team was checking a high cpu alert on a payment system cluster. He found some PIDs that were consuming high cpu and asked someone in the team on how to get more details. He was asked to use fuser to check. He did use the fuser but it was fuser -cuk. The -k for killing the process created quite a mess.

Re: Unix Admin Horror Story Summary (1992)

#75

Post by seqizz »

rilindo wrote: post #17
kloch wrote: post #3

I love this. It takes us back to a time when administering a Unix system was a Big Deal. Partly because they were rare and expensive. But also they were truly multi-user with dozens of people logged in at any given time.

They still are a big deal, only you can manage up to thousands of them and with modern automation, if you screw up one, you screw up all of them.

Yeah, we need a new page like this. But for "automation disasters".

Re: Unix Admin Horror Story Summary (1992)

#77

Post by krylon »

yeuxardents wrote: post #49
krylon wrote: post #32

Ironically, while I love Unix, I have spent most of my career shepherding Windows boxes. The only real horror story I got was a new coworker (turning me from "the IT guy" into "half the IT department") who looked through the Active Directory tree and found the GPO management part had replicated the organizational structure. Since there were no GPOs at the time, he considered this wasteful and confusing, so he went an…

This story worked out surprisingly well, usually, not so much (:

We were lucky restoring from backup worked, it had never been tested before. We were really lucky the CEO and the other coworkers took it so well.

And we were extremely lucky that the CEO was more interested in preventing such an outage in the future than in playing the blame game. I have a hunch this kind of situation can get really ugly once people start loosing their cool.

Re: Unix Admin Horror Story Summary (1992)

#78

Post by krylon »

geocrasher wrote: post #23

2003 or 2004. Customer called in and said that his dedicated server was hacked. I restored from backup. An hour later, he calls back. Hacked again . Restored again. An hour later, he calls back. He realizes that the hacker is him ! He's doing a thing, but doesn't know what he's doing wrong. So I have him email me the last thing he typed on his server, as root: rm -rf /home/user/path/to/thing /home/otheruser/path/to/s…

I've done something similar once, trying to remove backup files left by emacs:

  rm -f * ~
instead of

  rm -f *~
Is it weird to feel nostalgic about this kind of thing?

Re: Unix Admin Horror Story Summary (1992)

#79

Post by medstrom »

hnlmorg wrote: post #60
medstrom wrote: post #58

Earlier quoted context omitted.

Is there a rm variant that bundles an undo command?

There's a few ways to solve this: + a command that doesn't actually delete but rather moves the file to a trash folder that gets emptied periodically (this is exactly how a desktop environments work) + a command that removes the inode but leaves the data intact. This could get pretty messy pretty quickly on SSDs. + `rm` behaves as normal but you restore the file using forensic tools / file recovery tools. Similar pro…

There are a number of commands named "trash". But I don't want to keep in mind to execute "trash-empty" occasionally, especially when rm'ing the same large file several times (say, ones made by "dd if=/dev/zero ..."). So either a cron job for emptying the trash, or I think more elegantly, have "trash" just empty the trash before trashing the next file (i.e. there is only ever one thing in the trash).

Re: Unix Admin Horror Story Summary (1992)

#80

Post by medstrom »

mark-r wrote: post #70
geocrasher wrote: post #23

2003 or 2004. Customer called in and said that his dedicated server was hacked. I restored from backup. An hour later, he calls back. Hacked again . Restored again. An hour later, he calls back. He realizes that the hacker is him ! He's doing a thing, but doesn't know what he's doing wrong. So I have him email me the last thing he typed on his server, as root: rm -rf /home/user/path/to/thing /home/otheruser/path/to/s…

That was surprisingly subtle. I had to look at it about 3 times before I saw the problem.

The single slash was the first thing I saw, it jumped out to me before I saw anything else on the line. That's the value of experience.
Post reply on HN