Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

311–320 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#311
post #21

Not the worst at all, but probably one I found most amusing. One of my jobs included some sys admin tasks (this wasn't the position, but we all did dev ops), among my other responsibilities. I spent half a day going through everything with the person responsible for most of the admin tasks at the time. She was an extremely dilligent and competent admin, did absolutely everything through configuration management and k…

She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows I'm amazed that this is possible. How would I set something like that up? A realtime log of only the most significant events of a remote system? In fact, I'd like to take this opportunity of ignorance-admitting to ask the community for general linux/bsd sysadmin resources. What books should I read, or wh…

Check out www.sabok.org

Sys Admin Body of Knowledge

Re: Ask HN: What's the worst you've ever screwed up at work?

#312
post #271

Earlier quoted context omitted.

It sounds like you don't do continuous deployment. It divides the risk and improves agility. It pushes "change" to a non-management level.

"continuous deployment" is not a panacea to the issue. There's two primary ways of doing it, from what I understand: continuous deployment to a dev/test environment is the easiest for most organisations to move to. Due to the live environment being mission-critical, they can't afford the risk of any degradation of service. So you push regularly (after passing test suites) to a test environment, get a small subset of…

It helps a lot though. One of many small incremental changes, or one big monster change, which do you think is likely to break a system? And which is going to be easiest to diagnose and fix?

Re: Ask HN: What's the worst you've ever screwed up at work?

#313
post #166

Earlier quoted context omitted.

What position are you starting from? My old workplace was a university group where we (admins) were recruted from the available pool of PhD students. So I'm used to guiding people from "no knowledge" to "enough knowledge to be dangerous". The first step was to force the prospective admins to run a specific system on their "productive machine" and keep it in such conditions that _everything_ works. This way, a complet…

>If they were using Arch or Gentoo at home, If they are using gentoo, you should be finding someone else. Gentoo users are typically the most dangerous combination of profoundly ignorant, yet absurdly overconfident in their abilities. Seeing a bunch of autotools and gcc output scroll by does not teach you anything. But the mistaken reputation as an "advanced" distro makes people think that by using gentoo, they are t…

OK, it has been a few years since I tried, but if you could get Gentoo installed you must have known a fair amount.

Re: Ask HN: What's the worst you've ever screwed up at work?

#314

I love these topics. ~ 2007, working in a large bioinformatics group with our own very powerful cluster, mainly used for protein folding. Example job: fold every protein from a predicted coding region in a given genome. I was mostly doing graph analysis on metabolic and genetic networks though, and writing everything in Perl. I had a research deadline coming up in a month, but I was also about to go on a hunting trip…

My team did something similar once. We pushed a version to UAT with all the dev debug logging still turned on. It filled up a solaris disk so badly the SA had to get a bus to the data centre and fix it in person.

Re: Ask HN: What's the worst you've ever screwed up at work?

#315
post #307

Earlier quoted context omitted.

All true, but that still doesn't answer what possesses someone to pack their cell phone in a box .

>All true, but that still doesn't answer what possesses someone to pack their cell phone in a box. this is the primary difference I see between programmers and sysadmins, development and operations. I know programmers who don't own cellphones at all, while I know some sysadmins who take tertiary backup communication devices on vacation. It's a difference in focus. Of course, most programming jobs have /some/ operatio…

Note, I agree that packing your cellphone in a box was a mistake either way. But if you are primarily an operations/sysadmin type? that would be a really big deal kind of mistake, one that you probably wouldn't make very often. To a Developer type who saw their operations role as secondary, sure, it's still a mistake, but it's a smallish, forgettable kind of mistake.

Re: Ask HN: What's the worst you've ever screwed up at work?

#316
I performed a two-minute manual query on a MySQL database I was told was a backup. What I didn't realize was that it was a live backup, and that it would stop the production database from responding to queries for those two minutes, meaning that authentications failed for two minutes. Several colleagues called for my employment to be terminated immediately, but luckily they got distracted by other issues.

Re: Ask HN: What's the worst you've ever screwed up at work?

#317

Two screw ups come to mind. 1. First day at a job. I need to get familiar with a legacy system and get a SQL dump from it to create a local copy of the database. After some SSHing and MySQLing, I confuse my two split terminal panes and end up importing my local dump to production server. Of course database names and users were the same so I end up dropping the database. No biggie. Backups were available from previous…

This could be a strong example for why "the unix way" is better. You don't overwrite your server with anything unless you explicity rsync/scp it.

Re: Ask HN: What's the worst you've ever screwed up at work?

#318
I used to work at a major European telecoms company... Unlike everyone else I am not talking programming mistakes, these are generally physical fumbles:

One day I was doing a change control, I was scheduled to change some settings on the modulator of a satellite system providing internet access to a portion of the Middle East. I called the satellite operator and told them I wanted to do the scheduled work, as they would have to confirm I was outputting new configuration within the constraints of the contract.

I entered the change and re-initiated the modulation, the operative said he was seeing nothing. Now, because this signal was going to the middle east I couldn't see it in Europe and without substantial plumbing I couldn't tap in to the antenna. My heart started going, I was checking amplifiers, up-converters, everything and I couldn't see anything wrong. After a few minutes the client called panicking because the action should have taken moments, not minutes. After more confirming with the satellite company that I wasn't transmitting I checked back through my steps and eventually saw I had missed one crucial thing: when you changed certain parameters it muted the output! A quick few button presses and the patient man on the phone said "there it is!" and I can relax again.

Lessons learnt: 1) I should have noticed a critical LED on the modulator was not lit! 2) This is the reason change controls say "working period 10 minutes" (time taken 9m50s). 3) A good boss will defend you if you recognise your mistakes and don't f*ck up too badly. 4) Don't go for a quick drink with your brother before a night shift.

In the broadcast industry they say you aren't a real engineer until you've taken a TV network off air. Lets say I am very experienced, but my employers have never had a problem with me. This is probably because I have also seen people meltdown under the pressure of delivering live services to millions or even billions of viewers, but if you can keep cool you can deal with it.

Re: Ask HN: What's the worst you've ever screwed up at work?

#319
post #79

Earlier quoted context omitted.

In the past, I've set up big MOTD style messages that say "PROD" in fancy ASCII graffiti when I ssh/connect a DB client/whatever to production. I think I will set one of those up now for my current setup. Also, sort of related, I'm using MacOS, and in the back of my head I've wanted to create a tool that will change the color of the menu bar (at the top of the screen) to, say, bright yellow, when I'm connected to the…

It doesn't really solve anything, but I've done bright blue prompts for staging, red for production, and green for development.

This is essentially what I do - Black on White for production, White on Black for development. If I'm running development commands on a Black on White screen, something doesn't feel right. It isn't a life-or-death application, so this is enough.

Re: Ask HN: What's the worst you've ever screwed up at work?

#320

Earlier quoted context omitted.

Who ships a product release while in no position to monitor the results? Who ships a product release immediately after a commit?

"People who are shipping actual products instead of talking about them on message boards", perhaps. Running someone else's product company from a message board is a little like playing Jeopardy! from your couch, right?

You're right, I just run my own product company from my couch, and we'd never be daft enough to ship something critical right before disappearing into the ether.
Post reply on HN