Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

141–150 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#141
post #31

I wrote a piece of code controlling an assembly line machine. These machines require manual operation, and would come with a light curtain, which detects when someone places their hand near the moving parts, and should temporarily stop the machine. A relatively minor bug in the software that I wrote caused the safety curtain to stop triggering when a certain condition was met. We discovered this bug after an operator…

Couldn't read your comment without a shudder and my brain going straight to the Therac 25 incident(s).

I hadn't heard about this so I looked it up. Chilling.

http://en.wikipedia.org/wiki/Therac-25

Re: Ask HN: What's the worst you've ever screwed up at work?

#142
post #79
post #58

Earlier quoted context omitted.

Sure there's plenty of blame to spread around, but I still would have felt terrible if someone had been hurt or killed. What system would you put in place to prevent this? The issue was that I connected to prod when I thought I was connecting to a test DB. We each had different credentials for prod vs everything else, but the SQL client remembered my username and password. Anyone with prod access could have made the…

In the past, I've set up big MOTD style messages that say "PROD" in fancy ASCII graffiti when I ssh/connect a DB client/whatever to production. I think I will set one of those up now for my current setup. Also, sort of related, I'm using MacOS, and in the back of my head I've wanted to create a tool that will change the color of the menu bar (at the top of the screen) to, say, bright yellow, when I'm connected to the…

Maybe you could set a translucent menu bar, then script something to change the top 22px of the desktop background based on the VPN connection status. It's hacky, but it'd work.

Another option would be to configure your routes. At a previous job, I set up my home router to connect to the VPN and route 10.* to the VPN interface. Setting this up isn't easy, but it's oh-so-convenient. Reading http://wiki.openwrt.org/doc/howto/vpn.client.pptp will start you on the right track.

Be careful though. This gives anyone on your home network access to work. It almost certainly violates security policy. I only did this because I knew I'd just be chastised if I got caught. (Same goes for running rogue APs at work.)

Re: Ask HN: What's the worst you've ever screwed up at work?

#143
I can't decide between these two:

1) after few months working in a bank, I was doing some simple admin check task via RDP to a Windows 2003 (no, maybe 2000) server, when I right-clicked the network icon and instead of clicking the properties options i clicked "disable". Just the time to say "oh sh!t" and to realise that it was the production Trading On Line machine, on a remote datacenter, during market hours, and to discover couple of minutes later that the KVM over IP was crappy and was not working. We had to call the datacenter operators to go back to the local KVM and re-enable the NIC.

Lesson 1: Better move slowly when you're on a production machine (and also have plan B and C to reach your machines is a good idea)

2) same bank, one or two years later, I was doing some testing on a new mail system that integrated also VoIP (SIP). Mail/SIP System running in a VM (I think Vmware Server at that time) in the same remote datacenter as above. So, I enable the SIP feature and after few seconds, bum, we lose the whole (production) datacenter and the connection between the local server room and the datacenter. Panic, I look at my colleague, WTF in stereo, everything come back for few sec, bum again down. Long story short, the issue was that that version of Netscreen firewall ScreenOS had a buggy ALG implementation for SIP that lead to core dumps. The fun thing is that we had two of those in HA, same version of course, so they were bouncing between core dumping, rebooting slave becoming master and then core dumping again etc.. We had to ask a datacenter operator to reach the rack, disconnect one of the cables from the firewall (the one that was managing the traffic of the DMZ where that machine was hosted) and then reach the virtual host to kill the machine.

Lesson 2: you can segment your network but if everything is connected through the same device(s), sh!t can still hit the fan...

Re: Ask HN: What's the worst you've ever screwed up at work?

#144
post #22

One summer in college, I got an internship at a company that made health information systems. After fixing bugs in PHP scripts for a couple weeks, I was granted access to their production DB. (Hey, they were short on talent.) This database stored all kinds of stuff, including the operating room schedules for various hospitals. It included who was being operated on, when, what operation they were scheduled for, and im…

Uh, has anyone on this thread heard of HIPAA? I'm pretty sure having a summer intern get full access to actual patient data shouldn't be possible under a properly implemented set of HIPAA processes, and the same goes for the accidental UPDATE.

The story reminds me of the day I was introduced to "BEGIN TRANS", "COMMIT" and "ROLLBACK" when someone upgraded the Sybase console and helpfully changed the default setting so we didn't need those pesky semi-colons to finish a query any more. The result was:

  DELETE * FROM TABLE x
  131054 rows deleted
  WHERE a = "foo"
  >> Malformed query 
Phone starts to ring a few seconds later as all the users saw their morning's work disappear.

This stuff is way too easy for us noobs. Thank goodness that with modern technology we've found ways to make sure it doesn't happen any more... :-)

Re: Ask HN: What's the worst you've ever screwed up at work?

#145
post #15

easy: me: "unix definitely won't just let me cat /dev/urandom > /dev/sda" other: "sure it will" me: what I learned? unix will absolutely let you hang yourself. 1998, production server for a fortune 5 company.

That's quite funny.

Linux will indeed let you hang yourself. I maintain that every experienced Linux user has seriously messed up their system at least once. More often personal / dev / test environment than production though...

Re: Ask HN: What's the worst you've ever screwed up at work?

#146
Long ago when I was, I think, a sophomore in college and worked for the university IT group, I was trying to add an external drive to an early NeXT machine [1]. I wanted to try out their fancy GUI development stuff, you see. I was at best a modestly competent Unix admin, and this was circa NextStep 1.0, so the OS was... rough. It was in the dark days of SCSI terminators, so just telling if the drive was properly connected and, if so, how to address it was challenging.

After a couple hours of swearing, instead of working from a root shell in my own account, I just logged into the GUI as root. And there was a pretty interface showing the disks. I could just click on one and format it. Hooray!

Well either the GUI was buggy or I clicked on the wrong disk, because as the format was going, I realized the external drive wasn't doing anything. I was formatting the internal boot hard drive. And since nobody but me gave a crap about this weird free box somebody had given them, they had repurposed it. As a file server. For the home directories of a bunch of my colleagues. Who were now collecting around me wondering what was going on. Oops.

No problem, says I. I'll just restore from backups. But this thing used a weird magneto-optical drive [2]. The only boot media we had was on an MO disk. The backups were on another. And there was only one of these drives, probably only one in the whole state. The drives were, of course, incredibly slow, especially if you needed to swap disks. Which, I eventually discovered, I would have to do about a million times to have a hope of recovery.

Long story short, I spent 28 hours in a row in that chair. It was my immersion baptism [2] in the ways of being a sysadmin. The things I learned:

Fear the root shell. It should be treated with as much caution as a live snake.

Have backups. People will do dumb things; be ready.

A backup plan where you have never tried restoring anything may lead to more excitement than you want.

Be suspicious of GUI admin tools. Avoid new GUI admin tools if at all possible. Let somebody else be the one to discover the dangerous flaws.

If you were smart enough to break something, you're smart enough to fix it. Don't give up.

When some young idiot fucks up, check to make sure that they are sufficiently freaked out. If they are, no need to yell at them. Instead support them in solving the problem.

Seriously, my colleagues were awesome about this. I went on to become an actual paid sysadmin, and spent many years enjoying the work. The experience taught me fear, and a level of care that sticks with me today. I'm sure at the time I was wishing somebody would wave a magic wand and make it the problems go away, but working through it gave me a level of comfort in apparent disasters that has been helpful many times since.

[1] http://en.wikipedia.org/wiki/NeXTcube [2] http://en.wikipedia.org/wiki/Magneto-optical_drive [3] http://en.wikipedia.org/wiki/Immersion_baptism

Re: Ask HN: What's the worst you've ever screwed up at work?

#147
post #21

Not the worst at all, but probably one I found most amusing. One of my jobs included some sys admin tasks (this wasn't the position, but we all did dev ops), among my other responsibilities. I spent half a day going through everything with the person responsible for most of the admin tasks at the time. She was an extremely dilligent and competent admin, did absolutely everything through configuration management and k…

She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows I'm amazed that this is possible. How would I set something like that up? A realtime log of only the most significant events of a remote system? In fact, I'd like to take this opportunity of ignorance-admitting to ask the community for general linux/bsd sysadmin resources. What books should I read, or wh…

Remote syslogging, generally.

There are any number of log tailers. One I've recently discovered and love is multitail by Folkert vanHeusden:

http://www.vanheusden.com/multitail/

Incidentally, a related utility (and one which works really well with multitail) is rsstail:

http://www.vanheusden.com/rsstail/

One option is to get an RSS/Atom feed of critical system events.

Re: Ask HN: What's the worst you've ever screwed up at work?

#149
Connected leads on an expensive piece of equipment, power live, being very careful with a pair of needle nose pliers. Because the power switch was way in the other room, tag out procedure took time and I was late.

Poof. Equipment electronics fried and useless.

I was chewed out. Could have been way worse.

Follow your safety procedures.

Post reply on HN