It is nice to see how replies have persons real name and organization in the header, something you rarely see in internet discussions today (FB being one big exception). Brings back memories of using usenet in the 90s.
Unix Admin Horror Story Summary (1992)
81–90 of 94 posts
Re: Unix Admin Horror Story Summary (1992)
#82This takes me back to roughly 1993. I was in a department running on a mix of Wyse green-screen terminals and, later, X terminals, when we got a budget upgrade that would roll out actual individual PCs -- 486s running SCO Open Desktop -- to everyone. (This was not cheap, it cost about £4000 for the hardware per seat, although the software was free because, er, this was back in the day when SCO was a respectable UNIX…
He knew just enough to shoot himself in the foot.
What makes you think $MANAGER was male?
Re: Unix Admin Horror Story Summary (1992)
#832003 or 2004. Customer called in and said that his dedicated server was hacked. I restored from backup. An hour later, he calls back. Hacked again . Restored again. An hour later, he calls back. He realizes that the hacker is him ! He's doing a thing, but doesn't know what he's doing wrong. So I have him email me the last thing he typed on his server, as root: rm -rf /home/user/path/to/thing /home/otheruser/path/to/s…
In 1989 my then girlfriend now wife convinced me to go into contracting because it was 3x the pay vs working for our mutual employee, a savings and loan (sort of like a bank if you're not in the US) in receivership.*
Our mainframe guy had a buddy who said he knew unix so I interviewed him and rejected the guy because he clearly did not. The company hired the dude anyway and I went on to a fun contract with Shell where every server was named after mollusks which sounded amusing until you had to login to a box with a twenty letter name.
A few weeks after I left my girlfriend calls and to tell me the head of HR had come by and asked if had "done anything" to their computers? What? Like what? No, of course not. I called someone at the S & L to find Mr. New Unix admin had decided to "optimize" the cabling layout of the Sun servers which had in-rack storage. He didn't know that changing the cabling order changed device names, he didn't keep a map of device names to physical disks, which as a Sybase shop using raw disks was a big mistake. You had to know enough about Sybase to take it into maintenance mode and rename devices and if you got it wrong hello database corruption as the dbs spanned multiple disks. For the disks with mounted filesystems he didn't leave breadcrumbs on the root of each respective disk so he could mount it and figure out how to fix fstab. I don't think he even knew what fstab was. After breaking everything he claimed I had somehow dialed in and my "hacking" was to blame.
*I wasn't angry or upset at the S & L, but it was a valuable lesson for me. Hired as a 'C' programmer to write mortgage bond software when the company discovered I had database skills they also had me write a payroll app, which is how I knew everyone's salary and that the previous sysadmin I worked for made $10k/yr more than me. When I asked for a raise to his pay after officially taking over his job (which I had effectively been doing anyway) I got like a $3k raise and the employer claimed it was the best they could do because of their receivership situation.
That was the impetus of looking for another job because my gf was like if you can get more you need to do it. Fiiiiine. When I turned in my notice, my boss' exact words were "Oh, shit" because he knew he'd been underpaying me and there was no replacing a kid making $24k a year who was still coding mortgage analysis software, acting as their DBA and sysadmin. Even if I was the type to exact some kind of revenge (I was not) the fact they had to hire two or three people to replace me was enough. When my boss magically had money for a raise, thanks to my aforementioned payroll app I got to say one of the most satisfying things I've ever said to anyone: "Vic, I make more than you do, now. What's your counter?" Sorry if that's a pointless, rambling sorry. The nightmare was getting accused of hacking some company because of a mishire.
Supporting traders at Enron Capital & Trade was fun because the traders had root to their SPARCstations. In fact the interview was to fix an unbootable SPARCstation with a compressed kernel, something the support team had happen to them when a user needed more disk space. You had to know Solaris well enough to have the installboot syntax memorized and those workstations were so slow to boot you essentially got one shot at it because it was a timed test.
Re: Unix Admin Horror Story Summary (1992)
#84Earlier quoted context omitted.
Single-sided: 360 KB, double-sided (or double density): 720 KB, high density: 1.44MB. There were 2.88 MB disks and drives but they never gained much traction, because they were expensive and the PC industry kept promising various "floppy killers" like the ZIP drive.
The superdisk were what I wish had caught on. They could read HD floppies (though not any 2.88MB formats) and at some iteration were able to treat a single HD floppy as a 32MB(!!) tape drive. IOMega seemed to have better marketing though.
I wish Sony had made a serious effort to make MiniDisc a data format. They did make data drives, but they were slow, expensive, and barely available.
140 MB data (in 1993!). Nearly infinite rewrites. Nature of the medium is that bit rot is almost impossible without direct physical damage. I enjoyed them in the twilight of their existence, just before MP3 players started to have reasonable storage capacities and prices (mine were/are all used). And the ones I recorded in 2002 still sound just fine if I play them today. It was a great replacement for cassettes; too bad it never caught on in the US.
Re: Unix Admin Horror Story Summary (1992)
#85Earlier quoted context omitted.
The superdisk were what I wish had caught on. They could read HD floppies (though not any 2.88MB formats) and at some iteration were able to treat a single HD floppy as a 32MB(!!) tape drive. IOMega seemed to have better marketing though.
I wish Sony had made a serious effort to make MiniDisc a data format. They did make data drives, but they were slow, expensive, and barely available. 140 MB data (in 1993!). Nearly infinite rewrites. Nature of the medium is that bit rot is almost impossible without direct physical damage. I enjoyed them in the twilight of their existence, just before MP3 players started to have reasonable storage capacities and price…
FWIW Zip was only 1 year later with 100MB (and IMO less durable media). I agree that MD was better, but Sony in the 90s didn't seem to want to do anything that they couldn't completely own themselves.
My recollection is that Zip media was also way cheaper than blank MDs, but I don't have any sources for that.
Re: Unix Admin Horror Story Summary (1992)
#86Earlier quoted context omitted.
That was surprisingly subtle. I had to look at it about 3 times before I saw the problem.
The single slash was the first thing I saw, it jumped out to me before I saw anything else on the line. That's the value of experience.
I do not envy you your experience.
Re: Unix Admin Horror Story Summary (1992)
#87The best thing about these, for me, is the nostalgia over the email addresses. So many domains I recognize from back in the day.
Re: Unix Admin Horror Story Summary (1992)
#88Earlier quoted context omitted.
The single slash was the first thing I saw, it jumped out to me before I saw anything else on the line. That's the value of experience.
I do not envy you your experience.
It's something you learn just because of this kind of experience. For me, it was my attention to detail that spotted it immediately, and the customer facepalmed hard. It's one of those things that is easier for a second party to spot. It's why I read everything that's critical out loud before hitting enter, or publishing, or whatever.
Re: Unix Admin Horror Story Summary (1992)
#89I had a contractor installing software last week. He was granted full sudo permissions despite having demonstrated a unique set of command line skills. Everything was going fine until I get a text from his manager saying the contractor couldn't SSH in this morning. Turns out he had gotten frustrated with some file/folder permissions in the directory where he was supposed to install the software package. So he simply…
Why was he not fired?
Re: Unix Admin Horror Story Summary (1992)
#902003 or 2004. Customer called in and said that his dedicated server was hacked. I restored from backup. An hour later, he calls back. Hacked again . Restored again. An hour later, he calls back. He realizes that the hacker is him ! He's doing a thing, but doesn't know what he's doing wrong. So I have him email me the last thing he typed on his server, as root: rm -rf /home/user/path/to/thing /home/otheruser/path/to/s…
This is precisely why I always `-v` when `rm`ing recursively. It might be closing the barn door after the proverbial horse has bolted; but at least the fuck up is visible and in some circumstances you have a fighting chance to kill `rm` before too much damage has been done.
How about just don't use -f all the damn time?