Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix Admin Horror Story Summary (1992)

www-uxsup.csx.cam.ac.uk

Post reply on HN
Print view View original

21–30 of 94 posts

Re: Unix Admin Horror Story Summary (1992)

#21

Post by stryan »

DonHopkins wrote: post #12

I posted this horror story before, with a link to Pete "Gymble Roulette" Cottrell's infamous contest at the end (which I wasn't supposed to tell anyone outside of UMD CS Dept staff about): https://news.ycombinator.com/item?id=15802533 Pyramid's OSx version of Unix (a dual-universe Unix supporting both 4.xBSD and System V) [1] had a bug in the "passwd" program, such that if somebody edited /etc/passwd with a text edit…

What a small world; I read this comment while sitting in the UMD CS department machine room.

Glad bad login programs aren't something I have to deal with anymore (knock on wood).

Re: Unix Admin Horror Story Summary (1992)

#22

Post by qwertox »

This reminds me of the time when I execuded `rm -rf ~` in hopes of deleting an erroneously created directory.

Or when my "wrongly" expanded `mv` command moved all the files and directories in home into the last directory of the home directory. Which was a NTFS mount, leading to a loss of all the file permissions.

Re: Unix Admin Horror Story Summary (1992)

#23

Post by geocrasher »

2003 or 2004. Customer called in and said that his dedicated server was hacked. I restored from backup.

An hour later, he calls back. Hacked again. Restored again.

An hour later, he calls back. He realizes that the hacker is him! He's doing a thing, but doesn't know what he's doing wrong. So I have him email me the last thing he typed on his server, as root:

   rm -rf /home/user/path/to/thing /home/otheruser/path/to/somethingelse / home/path/to/some/other/thing/altogether

Re: Unix Admin Horror Story Summary (1992)

#25

Post by drewg123 »

pjmlp wrote: post #16
drewg123 wrote: post #11

Earlier quoted context omitted.

I remember being assigned to look into Solaris when working as a volunteer sysadmin in grad school, where we were a SunOS shop. I took a sparcstation, wiped it, and installed Solaris. This was 1992 or so, so it must have been 5.0 or 5.1. I hated it, but I don't remember very many specifics about why I didn't like it. I think it was partially the unbundled compilers, combined with everything just being "different", co…

That was around the time that GCC finally started to get some wind, due to the unbundling of UNIX SDK.

I think it also got wind because it was just so much easier to compile stuff with gcc than it was to use the vendor compilers, with all of their incompatible flags and extensions. This was in the days before package managers, when everybody compiled open source stuff themselves, a lot of things didn't use autoconf, etc.

I remember compiling almost all open source stuff (emacs, tex, postscript, file utils, etc) with GCC, and reserving the vendor compiler for situations where performance actually matters (math / linear algebra packages, professors' code).

EDIT: I remember a few years where people tended to assume all the world ran SunOS 4.1, just like people assume all the world runs some flavor of debian/ubuntu now.

Re: Unix Admin Horror Story Summary (1992)

#26

Post by terr-dav »

geocrasher wrote: post #23

2003 or 2004. Customer called in and said that his dedicated server was hacked. I restored from backup. An hour later, he calls back. Hacked again . Restored again. An hour later, he calls back. He realizes that the hacker is him ! He's doing a thing, but doesn't know what he's doing wrong. So I have him email me the last thing he typed on his server, as root: rm -rf /home/user/path/to/thing /home/otheruser/path/to/s…

I wonder if there's there a terminal setting or font that renders whitespace like a ]-shaped underscore? Could solve a whole class of bugs.

Re: Unix Admin Horror Story Summary (1992)

#27

Post by jcynix »

patrickdavey wrote: post #18
whartung wrote: post #10

Worst thing I ever did was cross hard mount NFS volumes across two machines. With a hard NFS mount, the mount will hang until the other machine responds. When we had to power cycle the two servers, they would not come up as they were deadlocked waiting for each other. That was exciting.

How did you fix it?

Been there, done that. Ok, a colleague did it, sbkut 30 years ago ;-)

IIRC resetting the machine and forcing it to single boot was the solution.

Re: Unix Admin Horror Story Summary (1992)

#29

Post by treesknees »

whartung wrote: post #10

Worst thing I ever did was cross hard mount NFS volumes across two machines. With a hard NFS mount, the mount will hang until the other machine responds. When we had to power cycle the two servers, they would not come up as they were deadlocked waiting for each other. That was exciting.

This is somewhat common in environments with stable power, you basically never have the entire IT system go down and come back up at the same time.

Re: Unix Admin Horror Story Summary (1992)

#30

Post by cf100clunk »

butterfi wrote: post #7

I thought I would find these funny and instead they just made me anxious. I mean, they are funny, I guess I just have PTSD from years of Unix administration.

Came here to say the same thing... Unix sysadmin since 1988. I would laugh about these stories but I just cannot without a lump in my throat and a few mea maxima culpas in my own mind. Reminds me of having heard somewhere that Tom Waits, on watching the litany of road show disasters parodied on This Is Spinal Tap, wept rather than laughed.

My own worst Unix Admin Horror Story is a variant of the classic ''accidental delete-restore from backup if you've got one'' scenario: in the early '90s I accidentally repopulated all YP tables on a production Sparcstation 10 machine in real time on a busy workday, but the tables had not been kept up to date! It took until the next day to restore from backup and a further day of research and testing to get all the YP tables up to proper state, then write scripts to keep them updated. (This was before Sun was legally forced to rename YP to NIS, btw).

Post reply on HN