Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#71

Post by icedchai »

gargarplex wrote: post #27
michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

I always add a LIMIT even when not necessary. Why doesn't MySQL have a version control baked in? Even if it preserves just the last n hours of state..

It kind of does. It's called the binary log.

Re: Ask HN: What's the worst you've ever screwed up at work?

#72

Post by DCoder »

gargarplex wrote: post #27
michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

I always add a LIMIT even when not necessary. Why doesn't MySQL have a version control baked in? Even if it preserves just the last n hours of state..

Since you're using MySQL, you can start it with --i-am-a-dummy [1]. Not a backup, but still handy.

[1] http://dev.mysql.com/doc/refman/5.6/en/mysql-tips.html#safe-...

Re: Ask HN: What's the worst you've ever screwed up at work?

#73

Post by kps »

Robin_Message wrote: post #56
dllthomas wrote: post #51

Earlier quoted context omitted.

' And guess what's the value of "HOME" in bash?' In the rm line of the snippet above, "/some/location". Magic variables in bash tend to lose their magic once set.

I assume `set HOME = /some/location` is the tcsh syntax to set a variable. In Bash, it doesn't do anything useful.

  > In Bash, it doesn't do anything useful.
In sh and derived shells, it sets the arguments ($1, $2, and so on). In this case you end up with $1 being ‘HOME’, $2 being ‘=’, and $3 being ‘/some/location’.

Re: Ask HN: What's the worst you've ever screwed up at work?

#74

Post by unfunco »

I think everybody has done this at some point, and I'm sure I will not be the last person to have done it; leaving the WHERE clause from DELETE and UPDATE statements when writing SQL, I caused about 45 minutes downtime on our RDS instance the last time I did it, but since we had multi-AZ setup, no data was lost. I also frequently get mixed up between development and production environments.

Every database alias I have now has the MySQL --i-am-a-dummy flag appended. This has been a career-saver in my eyes.

Re: Ask HN: What's the worst you've ever screwed up at work?

#76

Post by glogla »

michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

This is why, while I hate Oracle and everything they represent as a company, I kind of like their database because of the flashback feature. You can do

  SELECT * FROM table AS OF TIMESTAMP some_timestamp;
and that is pretty practical. It works online, no restore, no nothing, and while it only works as long as the old data are in logs, on a production system, you should have the spare space to have some history. Theres also FLASHBACK TABLE tab to BEFORE DROP but that shouldn't happen, right?

Of course, you should probably do every update of production data in a transaction, check the result and then commit, and if you want to be sure, you can do UPDATE ... RETURNING to check what's changing. Autocommit on manual access to production is pretty crazy. But still, flashback is useful.

Re: Ask HN: What's the worst you've ever screwed up at work?

#77

Post by edw519 »

Boss: We have thousands of bad orders that must be fixed now!

Me: No we don't. We have 121 bad orders.

Boss: There are thousands of them!

Me: No there aren't. There are exactly 121 of them. I'm sure.

Boss: I'm not going to argue with you!

Me: Good. Because you'd lose.

I fixed 121 orders that night. The next day my login & password wouldn't work.

Re: Ask HN: What's the worst you've ever screwed up at work?

#78

Post by ixmatus »

sillysaurus2 wrote: post #69
yan wrote: post #21

Not the worst at all, but probably one I found most amusing. One of my jobs included some sys admin tasks (this wasn't the position, but we all did dev ops), among my other responsibilities. I spent half a day going through everything with the person responsible for most of the admin tasks at the time. She was an extremely dilligent and competent admin, did absolutely everything through configuration management and k…

She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows I'm amazed that this is possible. How would I set something like that up? A realtime log of only the most significant events of a remote system? In fact, I'd like to take this opportunity of ignorance-admitting to ask the community for general linux/bsd sysadmin resources. What books should I read, or wh…

Papertrail is great for this...you can of course setup syslog to route to a central server and just be logged in tmux / screen on that particular machine to read off the stream of logs (I prefer papertrail though + saved searches and hipchat "pings" when saved searches are matched on incoming events).

General devops / sysops/ sysadmin knowledge can be had through a variety of means - I got most of my knowledge from simply reading the FreeBSD manual and making a lot of mistakes with my own servers.

Re: Ask HN: What's the worst you've ever screwed up at work?

#79

Post by canadev »

ggreer wrote: post #58
munificent wrote: post #50

Earlier quoted context omitted.

You didn't screw up here. The entire infrastructure, org chart, and policies that allowed you to accidentally modify a production database containing critical medical information screwed up. Blaming yourself here is like blaming yourself for being hurt after being told to drive a car with no seatbelt or brakes.

Sure there's plenty of blame to spread around, but I still would have felt terrible if someone had been hurt or killed. What system would you put in place to prevent this? The issue was that I connected to prod when I thought I was connecting to a test DB. We each had different credentials for prod vs everything else, but the SQL client remembered my username and password. Anyone with prod access could have made the…

In the past, I've set up big MOTD style messages that say "PROD" in fancy ASCII graffiti when I ssh/connect a DB client/whatever to production. I think I will set one of those up now for my current setup.

Also, sort of related, I'm using MacOS, and in the back of my head I've wanted to create a tool that will change the color of the menu bar (at the top of the screen) to, say, bright yellow, when I'm connected to the VPN so that I don't accidentally visit a porn site while still connected to work.

That said, neither of these systems is even close to fail-proof :)

Re: Ask HN: What's the worst you've ever screwed up at work?

#80

Post by sentenza »

sillysaurus2 wrote: post #69
yan wrote: post #21

Not the worst at all, but probably one I found most amusing. One of my jobs included some sys admin tasks (this wasn't the position, but we all did dev ops), among my other responsibilities. I spent half a day going through everything with the person responsible for most of the admin tasks at the time. She was an extremely dilligent and competent admin, did absolutely everything through configuration management and k…

She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows I'm amazed that this is possible. How would I set something like that up? A realtime log of only the most significant events of a remote system? In fact, I'd like to take this opportunity of ignorance-admitting to ask the community for general linux/bsd sysadmin resources. What books should I read, or wh…

What position are you starting from? My old workplace was a university group where we (admins) were recruted from the available pool of PhD students. So I'm used to guiding people from "no knowledge" to "enough knowledge to be dangerous". The first step was to force the prospective admins to run a specific system on their "productive machine" and keep it in such conditions that _everything_ works.

This way, a complete admin newbie would learn about digging through the systems by working out the kinks of practical everyday problems. Remember, this is only the most basic instruction, nowhere near enterprise-grade.

If there was a "prospective admin" who had never before run Linux, I'd tell them to install and use Ubuntu/Mint. (Those guys whould usually only be trained to be a helping hand for a "senior" admin.)

If he'd already used Ubuntu at home, I'd tell them to start using Debian and work out how to set up an SSH server and set up their home machine so they could access it remotely.

If they had dabbled with Debian, Fedora, SuSe or something similar, I'd tell them to install Arch and set up some "interesting things", like a mail server or a nis server.

If they were using Arch or Gentoo at home, I'd just personally show them the important things about our system and have them wingman with me for a few days.

If you are already an advanced Linux or BSD user, my approach is of course not appropriate. Instead I'd recommend to pick skills that you want to learn (iptables? Exim?) and set that up. Read Manuals! Read RFCs!

Best of luck.

Post reply on HN