Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

Post reply on HN
Print view View original

151–160 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#151

Post by dllthomas »

hapless wrote: post #102
dllthomas wrote: post #51

Earlier quoted context omitted.

' And guess what's the value of "HOME" in bash?' In the rm line of the snippet above, "/some/location". Magic variables in bash tend to lose their magic once set.

nope. try it: set HOME; echo $HOME. on IRIX, $HOME for root is almost certainly root. So the resulting command is rm -rf /

I missed that set HOME doesn't set HOME (I was pre-coffee).

What I said about un-magic-ing is true for sure. "HOME=foo; echo $HOME" will print "foo".

Interestingly, it's propagated. "HOME=foo sh -c 'echo $HOME'" will also print "foo".

Re: Ask HN: What's the worst you've ever screwed up at work?

#153

Post by wpietri »

ggreer wrote: post #58
munificent wrote: post #50

Earlier quoted context omitted.

You didn't screw up here. The entire infrastructure, org chart, and policies that allowed you to accidentally modify a production database containing critical medical information screwed up. Blaming yourself here is like blaming yourself for being hurt after being told to drive a car with no seatbelt or brakes.

Sure there's plenty of blame to spread around, but I still would have felt terrible if someone had been hurt or killed. What system would you put in place to prevent this? The issue was that I connected to prod when I thought I was connecting to a test DB. We each had different credentials for prod vs everything else, but the SQL client remembered my username and password. Anyone with prod access could have made the…

In a case like that, where clicking on the wrong thing could result in death? I would never allow production database access for anything other than the running app.

I'd have an emergency procedure, sure, one where in some dire circumstance somebody could poke a hole in the firewall, change the database configuration, open a sealed envelope, and then look at/change the real data.

But in normal circumstances, anybody who really needed to see prod data would look at a read-only copy. (Or better, would look at an identity-scrambled version of it.) Any anybody who needed to change it would write a bit of code to do the work and take it through the normal review and push process.

Re: Ask HN: What's the worst you've ever screwed up at work?

#154

Post by arethuza »

I led an engineering team that almost sent out a demo on tens of thousands of IBM CDs (this was 1998) that contained test data that included some that had been sourced from the worst possible alt.* newgroup.

As it turned out the only data that did go out was the single word "sheep" in the search index.

Re: Ask HN: What's the worst you've ever screwed up at work?

#155

Post by CurtMonash »

Short version:

I was a stock analyst, for a firm with dozens of institutional salesmen and thousands of retail brokers. Some of my recommendations were very, very wrong.

The right thing to do is stand up, take the heat, and explain what you now know as best you can. I learned that watching a colleague who I thought was otherwise an unserious ass.

Re: Ask HN: What's the worst you've ever screwed up at work?

#156

Post by rmchugh »

ggreer wrote: post #58
munificent wrote: post #50

Earlier quoted context omitted.

You didn't screw up here. The entire infrastructure, org chart, and policies that allowed you to accidentally modify a production database containing critical medical information screwed up. Blaming yourself here is like blaming yourself for being hurt after being told to drive a car with no seatbelt or brakes.

Sure there's plenty of blame to spread around, but I still would have felt terrible if someone had been hurt or killed. What system would you put in place to prevent this? The issue was that I connected to prod when I thought I was connecting to a test DB. We each had different credentials for prod vs everything else, but the SQL client remembered my username and password. Anyone with prod access could have made the…

In my work, developers only have read access to production servers (for checking logs etc). If you want to make a change to a production system, you need to go formally request it through OTRS. So this sort of situation can't really arise. You can of course still cock-up live systems through asking the sysadmins to do something stupid, but then the problem is stupidity, not carelessness.

P.S. You poor soul! I feel your pain.

Re: Ask HN: What's the worst you've ever screwed up at work?

#157

Post by georgemcbay »

wpietri wrote: post #146

Long ago when I was, I think, a sophomore in college and worked for the university IT group, I was trying to add an external drive to an early NeXT machine [1]. I wanted to try out their fancy GUI development stuff, you see. I was at best a modestly competent Unix admin, and this was circa NextStep 1.0, so the OS was... rough. It was in the dark days of SCSI terminators, so just telling if the drive was properly conn…

root shell, plus rm with any sort of wildcard matching, plus a bit too much of a delay before you get your shell prompt back results in a very specific kind of panicked anxiety that almost anyone who has been programming or sysadmining for a while can easily relate to.

Re: Ask HN: What's the worst you've ever screwed up at work?

#158

Post by Smerity »

I was testing disaster recovery for the database cluster I was managing. Spun up new instances on AWS, pulled down production data, created various disasters, tested recovery.

Surprisingly it all seemed to work well. These disaster recovery steps weren't heavily tested before. Brilliant! I went to shut down the AWS instances. Kill DB group. Wait. Wait... The DB group? Wasn't it DB-test group...

I'd just killed all the production databases. And the streaming replicas. And... everything... All at the busiest time of day for our site.

Panic arose in my chest. Eyes glazed over. It's one thing to test disaster recovery when it doesn't matter, but when it suddenly does matter... I turned to the disaster recovery code I'd just been testing. I was reasonably sure it all worked... Reasonably...

Less than five minutes later, I'd spun up a brand new database cluster. The only loss was a minute or two of user transactions, which for our site wasn't too problematic.

My friends joked later that at least we now knew for sure that disaster recovery worked in production...

Lesson: When testing disaster recovery, ensure you're not actually creating a disaster in production.

Re: Ask HN: What's the worst you've ever screwed up at work?

#159

Post by BU_student »

patio11 wrote: post #93

I've only cried literal tears once in the last ten years, over business. Due to inattention while coding during an apartment move, I pushed a change to Appointment Reminder which was poorly considered. It didn't cause any immediate problems and passed my test suites, but the upshot is it was a time bomb that would inevitably bring down the site's queue worker processes and keep them down. Lesson #1: Don't code when y…

> Fortuitously, AR didn't have all that many customers at the time, so only 15 or so people were affected. Less than fortuitously, those 15 folks had 10 to 100 messages queued, each.

Excuse me for caviling at your post, but "fortuitously" is a synonym for "accidentally", not "fortunately".

Re: Ask HN: What's the worst you've ever screwed up at work?

#160

Post by dllthomas »

nextweek2 wrote: post #135
d0m wrote: post #129

Earlier quoted context omitted.

Can't agree more with Patrick, mine is formulated a bit differently: Lesson #1: Don't push code on Friday afternoon. Lesson #2: Beer, Code and Commit is totally fine. Just don't push! Wait until next day to review and push/deploy it..

This is the difference between a software developer and a senior software developer. I never release anything on a Friday OR the day before a holiday. ALWAYS plan it afterwards, give any technical reason you like (solar flares). Project managers will realise and thank you for it in the long run.

This is definitely something I value about working in finance. Exchanges (the ones we're caring about anyway) are all closed over the weekends.
Post reply on HN