Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

191–200 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#191
I had installed "osCommerce - Open source E-Commerce platform" just like Magento on one of our client who had > 500 transactions a day.

Some how in settings, we had flag "Store Credit Card Info" as "Plain Text" enabled. The Admin/Staff of that client could have use this information to make transactions (As in Backend it would show Full CC info into order details)

We didnt realized untill we worked on it again for some bug fixes and adding new features.

Lesson Learned :- When transitioning from DEV to PROD env, make sure to check all these critical flags and correctly set

Luckily, the client didnt had any idea about what was wrong in backend.

Re: Ask HN: What's the worst you've ever screwed up at work?

#192
sudo chown -R myname:myname /

Learned: Learning on the job as you hack away on problems is great, but recognize that it's one part enthusiasm and one part risk management. Also learned to never try anything on the command line that wouldn't want to see pulled from my bash history and stuck on the breakroom fridge. Also learned to cope with humiliation well.

Re: Ask HN: What's the worst you've ever screwed up at work?

#193
post #89

I once revoked my bosses e-mail and VPN access because his password was 'password123'. It was my job to keep things safe after all and I had asked him nicely a few times. EDIT: I proposed a new password of: @$tevezA$$ignedPwD@# (Steve's Assigned Password) He said no to that one.

Heh, I have a co-worker that recently deployed a production router with a blank administrator password. Fortunately, I've got automated jobs in place that find those kind of things. The password is now a variation of "John is a dumbass" and I'm just waiting for him to ask for the password.

That's awesome!

Re: Ask HN: What's the worst you've ever screwed up at work?

#194

# cd /etc # emacs inetd.conf # ls ... ... inetd.conf ... inetd.conf~ ... # rm * ~ # ls # ls

Man, for some reason that double 'ls' made me laugh for 5 minutes. Just tried to imagine the surprised look on your face.

irl, that would have been:

  $ ls
  $ sleep 5
  $ ls

Re: Ask HN: What's the worst you've ever screwed up at work?

#195
I messed up epically on an interview. It was a 3 part interview for a JS/RoR coder.

1. I passed the resume and chat portion

2. I passed the telephone questionnaire and got along great with the interviewer

3. (Fail) I scheduled my interview on a Friday at 4:30pm and there is a 30 min travel time. I left 1hr early...still it was Memorial Day weekend, so I thought the streets would be quicker than the freeway since it was at a stand still. I was so stressed that I literally had an anxiety attack and couldn't even find the address. Never happened to me before, so I'll never forget it.

Re: Ask HN: What's the worst you've ever screwed up at work?

#196
My worst would have been catastrophic if I had waited one minute to make my mistake.

I was commissioning a new control system at a power plant's water treatment facility. I was fairly new to the industry and had mostly looked over the guy who did the bulk of the work's shoulder as on the job training.

This particular day the guy was out sick and we had to finalize a couple of things before we ran through the final tests.

There was an instruction to open a valve to fill a tank and it had the wrong variable linked to it. The problem was to maintain the naming standards I had to do a download to the processor to make the change. When I had been doing work in the office this was not a big deal, download the program to the processor, it stops running for a moment while it loads the new logic into memory and starts back up.

Not thinking through the implications of the processor shutting down while the process was up and running I made the code changes, hit download and about 30 seconds later an operator came running over looking like he had seen a ghost and he was pissed.

While I was making my code changes the operator was hooking up a hose to drain a rail car of some chemicals. The way the valves were configured before I made my changes was correct and would have had no consequence it I didn't touch anything. The way the valves were configured when the processor restarted would have routed the rail car's contents to the wrong tank resulting in a reaction which would have created a huge plume of highly toxic gas. The way the wind was blowing this plume would have blown directly to the largest town in the area and could have killed a ton of people.

The operator heard the valves in question changing position before he opened the valve on his hose to empty the rail car and figured something was up. When he saw the whole process had shut down he got really angry because I had ignored the protocol in place to avoid such a disaster.

I got chewed out and kicked off the site. My boss attributed my mistake to inexperience and I had to give a safety presentation on what I did wrong.

Lessons learned: Be sure you are aware of any implications your actions have. If you are unsure or guessing about something stop what you are doing and go ask someone first.

Don't give people mission critical work on their first project and have them work unsupervised. Training is important.

Always be aware of safety requirements, especially when you are working with machinery, automated processes, chemicals or anything else that can hurt, maim or kill you.

Re: Ask HN: What's the worst you've ever screwed up at work?

#197
post #12

A local Subway franchise was the very first company that hired me. I was extremely young, shy, and intensely socially awkward, yet excited to join the workforce (as I had my eyes set on a Pentium processor). When I worked at Subway, the bread dough came frozen, but you would put loaves in a proofer, proof it for a certain amount of time, and then bake it. My first shift, however, got busy and I left several trays in…

Dear god the buns. Sympathize greatly with you there.

Re: Ask HN: What's the worst you've ever screwed up at work?

#198
Simple one really and probably most common. Realise a data integrity issue on DB, try to load from backups and notice that the backups have the same integrity issue. Find a backup from about 2 weeks previous where data is intact and piece together the good pages from the daily backups and the 2 weeks old one.

All in, took 4 days and a new server where the hard drive had stored bad pages on the DB. We lost 2 days of orders (they were processed through to the internal systems though so not really lost)

Lesson learned, validate backups and check page integrity when backing up

Re: Ask HN: What's the worst you've ever screwed up at work?

#199
post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

I did this once and have since always typed the WHERE first in an UPDATE.

UPDATE ... UPDATE WHERE ... [go back, go back] UPDATE SET ... WHERE ...

Re: Ask HN: What's the worst you've ever screwed up at work?

#200

Earlier quoted context omitted.

I always add a LIMIT even when not necessary. Why doesn't MySQL have a version control baked in? Even if it preserves just the last n hours of state..

It kind of does. It's called the binary log.

Didn't know about this! Thanks.

https://dev.mysql.com/doc/refman/5.0/en/binary-log.html

https://dev.mysql.com/doc/refman/5.0/en/point-in-time-recove...

Post reply on HN