Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unix Admin Horror Story Summary (1992)

www-uxsup.csx.cam.ac.uk

Post reply on HN
Print view View original

31–40 of 94 posts

Re: Unix Admin Horror Story Summary (1992)

#31

Post by csydas »

The first entry about adding tcsh is more or less the basis of one of the questions I use in technical interviews for our Linux team. It's less about specifics on tcsh and more just about explaining the hierarchy of linux/unix and why we have the bin/sbin directories under / and /usr; the more the candidate can explain (or even hypothesize), the more comfortable my team feels with their general curiosity/understanding of Linux/Unix.

It's a niche situation sure, but being able to understand the system and tooling you're working with to a degree to understand what options you really have shows a great deal of discipline and curiosity, for me at least. Again, it's less about "can you figure out what to do in this specific situation" and more "can you just explain what you look at every single day in plain and simple terms? Did you ever think about it?"

It's been a surprisingly revealing question for nascent Linux admins on how they react to questioning the things they look at every single day, and how ready they are to __really__ dig into the kernel internals.

Re: Unix Admin Horror Story Summary (1992)

#32

Post by krylon »

Ironically, while I love Unix, I have spent most of my career shepherding Windows boxes. The only real horror story I got was a new coworker (turning me from "the IT guy" into "half the IT department") who looked through the Active Directory tree and found the GPO management part had replicated the organizational structure. Since there were no GPOs at the time, he considered this wasteful and confusing, so he went and deleted it.

...

Except that what he did delete, it turned out, was the actual organizational structure of the Active Directory tree, including ALL user accounts. (It's hard to explain without visuals aids, the UI gave no indication it would delete the actual AD objects, not just the (non-existent) GPOs.)

Before long, people started calling to let us know they could no longer log into their computers or the terminal server. sigh It was a fairly stressful morning.

We really tried, for about 45 minutes, to resurrect the Active Directory tree, but it was no good (this was Windows Server 2008, so no AD Recycle Bin), so we had to restore the server from backup. I have since learnt that there is backup software that allows you to restore, say, your AD tree, or maybe even just a part of it, instead of the whole machine. Well, the backup software we had at the time suuuuucked, so not only did we have to restore the entire server, but we had to literally sit all day and watch the progress bar move at glacial speed.

In the end, we had the server up and running again, and fortunately both the company's CEO and most employees actually welcomed the opportunity to finally, FINALLY clean up their desks, something every single one of them had been delaying for a long time. And by the time we were done, I was just so exhausted I wasn't even mad at the newbie anymore.

At least we learnt from that mistake, though. Got ourselves a second domain controller, and a much better backup solution. In retrospect, I think it was probably a good thing - our boss took it with good humor, no data was lost, our backup system worked, but we also saw how badly it sucked, and the incident gave us some leverage to get the funding for said upgrades. Also, everyone had a clean desk, and since it was a Friday, a couple of coworkers decided to start their weekend early.

Re: Unix Admin Horror Story Summary (1992)

#33

Post by mdpye »

lifeisstillgood wrote: post #28

So much of this can be filed under "before we culturally accepted prod is different.."

Most of these stories relate to administering interactive multi-user machines, not the kind of thing we now think of as a server.

Users were simultaneously logged in at the shell going about their business in *nix, not sending stateless requests in to a server process.

And you certainly couldn't afford to have a duplicate of a machine that expensive.

The idea of multiple environments didn't really exist, and you mostly administered machines from within - hence many of the stories being about getting enough tools working again to straighten it out. You didn't have another machine (or perhaps the connectivity) to put the thing on the operating table from a working system.

Things were different...

Re: Unix Admin Horror Story Summary (1992)

#34

Post by oneweekwonder »

patrickdavey wrote: post #18
whartung wrote: post #10

Worst thing I ever did was cross hard mount NFS volumes across two machines. With a hard NFS mount, the mount will hang until the other machine responds. When we had to power cycle the two servers, they would not come up as they were deadlocked waiting for each other. That was exciting.

How did you fix it?

not op but we recently wanted to mount nfs and the sysadmin was adamant we use automounter[0] instead of fstab because if the nfs mount is not available it can hang the kernel.

Not sure if it is true or just sysadmin lore but was interesting enough to learn about a alternative.

[0]: https://linux.die.net/man/8/automount

Re: Unix Admin Horror Story Summary (1992)

#36

Post by hulitu »

rntksi wrote: post #2

>Well one time I was installing a minimal base system of Linux on a friends PC, so that we would have all the necessary utlitities to bring over the rest of the stuff. His 3 1/2 inch disk was dead, so when had to get the 5 1/4 inch version of the boot/root disk. Too bad that version, having to fit in 1.2M instead of 1.44, didn't have tar Heh ... I wonder how many years forward will people stop knowing what a 3 1/2 an…

Ah, the good old days. Single density, double density ( 720 kB, 1,44 MB). I heard also of 2.88 MB floppies - never saw one in real life. If i remember correctly 2.88 MB was double density double sided and you needed a special floppy drive.

Re: Unix Admin Horror Story Summary (1992)

#38

Post by karmakaze »

I really enjoy the recovery parts of the stories that have them, like a good Hollywood movie script, but real.

Unix wasn't very common after leaving university and I have more PC/LAN type stories. There was one memorable moment, where I was working very late in the office and got a call. [If working late, the main line would ring the entire office and I could press the blinky light to answer.] It was one of our consultants on the west coast who somehow had a corrupt filesystem, but that machine was the one that had all the project files for the many months of consulting work that the team had been developing. [I don't recall but it may have been CVS or SVN.]

The tricky bit was that it was using OS/2 and its HPFS filesystem so the usual file utilities wouldn't work. We had a number of IBM tech books on our bookshelves (because we also did mainframe consulting) and I'd been reading about terminal streams and one about the HPFS filesystem in particular. It mentioned boot blocks, superblocks, bands, allocation bit blocks, etc.

Being young (and dumb) went with "what's the worst thing that could happen" and came up with a plan: using the DOS 'nu' (Norton Utility) copy a few choice sectors from a similar spec-looking machine and try the OS/2 equivalent of 'chkdsk /f'--the client after all was IBM known for conformity. We first had to dial-up modem transmit the 'nu' program, but then we were coping the first 18 (or so) sectors to get the boot sector, partition table, boot program or other HPFS initial sector data; then there were some sectors in the middle of the disk that served as a kind of main description table with others in bands (that we didn't bother with). Guessed the starting point and number of sectors. This was a grasping at straws Hail Mary. Rebooted the machine, let the OS/2 run its chcdsk as it detected a problem, waited a long while until it was done. Unbelieveably it all worked! There might have been a couple open files lost and some files that were recently deleted being present, but no big differences. We didn't think we needed to tell anyone. He bought me beers as promised when I came to visit.

Bonus memory: LapLink with the parallel transfer cable was the shit in those days. https://en.wikipedia.org/wiki/LapLink_cable

Re: Unix Admin Horror Story Summary (1992)

#39

Post by bostik »

whartung wrote: post #10

Worst thing I ever did was cross hard mount NFS volumes across two machines. With a hard NFS mount, the mount will hang until the other machine responds. When we had to power cycle the two servers, they would not come up as they were deadlocked waiting for each other. That was exciting.

Speaking of NFS, ex-coworker had renamed his prior company's servers "notresponding" and "stilltrying".

The NFS client logs must have been glorious.

Post reply on HN