Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Times I've Messed Up as a Developer

medium.com

Post reply on HN
Print view View original

121–130 of 137 posts

Re: Times I've Messed Up as a Developer

#121

Post by Karupan »

During the beta for a product, we were working on sending out daily summary emails to our users. We had a cron job which should run at 8:00 AM in the morning which would process the queue and summarize all activities into a single email.

It being a very long day, I deployed the job with the cron expression * 8 * * * instead of 0 8 * * * , which meant it ran EVERY minute starting at 8:00 AM instead of just once.

The next morning I woke up to a flurry of text and emails saying customers had complained they received emails about a hundred times. Thankfully, we had a pause in the job which meant it went out only a hundred times and not thousands as one would have expected!

Re: Times I've Messed Up as a Developer

#122

Post by planteen »

Posibyte wrote: post #109
planteen wrote: post #105

Earlier quoted context omitted.

Did you have different credentials you were supposed to be using?

Afterwards, yes, but before that all staging tables belonged to a single role under a single staging superuser.

It doesn't seem like you really did anything wrong in that case.

Re: Times I've Messed Up as a Developer

#123

Post by foobarchu »

mmjaa wrote: post #74
mfontani wrote: post #57

Earlier quoted context omitted.

To prevent this, my iTerm2 has a custom background enabled when logged in on any production servers. I wish the same were easily possible on a Linux terminal without much "trickery" :|

Whats wrong with just setting PS1 to indicate that its a production server, and using a decent host naming scheme that properly indicates the function of the machine you're logged into? I mean this is what I do for my Linux machines, and I haven't deleted a Linux production machine in decades ..

Maybe this is an obvious question but...how do you do that if you ssh into your remote machines through a bastion that normalizes everyone to the same end user? I'm unable to set a custom .bashrc on the remote server, because its a shared user, so how would you change the ps1 on the remote box?

Re: Times I've Messed Up as a Developer

#124

Post by inertial »

seanwilson wrote: post #41
1ba9115454 wrote: post #2

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

This is why I do a "mv" to "tmp" instead of using "rm" when I can. Having special messages or colours to differentiate between production and other environments is a great idea as well. Generally if I'm about to do something on production, I'll close everything else to do with other environments as it's just too easy to slip up.

There's trash-cli that moves files to "trash". Alias "rm" to display an error message to avoid accidental deletions.

https://github.com/sindresorhus/trash-cli

Re: Times I've Messed Up as a Developer

#125

Post by rocky1138 »

donatj wrote: post #7

I was on my way out the door at my first job, and an SEO person stopped me and asked me if I could fix something. I just needed to delete a single row from a database. Easy enough. I get as far as “DELETE FROM table_name” and for some unknown reason run the damn thing without a WHERE. All the rows, gone! We did have nightly backups but the guy with access to them was notoriously hard to get the attention of and lived…

> We did have nightly backups but the guy with access to them was notoriously hard to get the attention of and lived in a different state.

Read up on "bus factor": https://en.wikipedia.org/wiki/Bus_factor

Re: Times I've Messed Up as a Developer

#126

Post by rocky1138 »

ChemicalWarfare wrote: post #37

Speaking of backups :) We had a very large mission-critical system that we developed and hosted for a fortune 50 client. Long story short - prod DB with tons of records took a crap, proper RCA was never done I suspected HW failure of some sort. The kicker was - the client grudgingly OKd the restore understanding that they will loose few hours worth of data. BUT. When a DBA attempted to restore - the backup was corrup…

The age-old story: if you don't test restore your backups, you don't have backups.

Re: Times I've Messed Up as a Developer

#127

Post by MaxBarraclough »

JoeMalt wrote: post #97
donatj wrote: post #52

Earlier quoted context omitted.

I'd say require a WHERE on deletes. If you want "WHERE true" that's fine, but require a WHERE.

If you start the MySQL Monitor with --i-am-a-dummy, this is enforced automatically

Interesting, thanks.

As far as I can tell, there's nothing like that in Postgres.

Re: Times I've Messed Up as a Developer

#128

Post by throwme211345 »

I once deleted the entire contents of /bin and /sbin on an HPUX box ~2004 with the HPUX servers origin circa 1996. Live ssh session: non-standard location. scp from a convenient near-replica and voila. Without the near-peer I would have quit and never went back to IT.

Re: Times I've Messed Up as a Developer

#129

Post by throwme211345 »

humblebee wrote: post #71
1ba9115454 wrote: post #2

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

One the last day of a colleague of mine, he decided to run `rm -rf --no-preserve-root /` to see what actually would happen on his workstation (it was going to be formatted shortly after). We were all standing around talking waiting for whatever would happen to happen before we saw him off. A few minutes in I asked if he had unmounted our NFS volumes. Nope. He quickly pulled the power, and that was that. I have no ide…

Oh fuck moment.

Re: Times I've Messed Up as a Developer

#130

Post by throwme211345 »

GlennS wrote: post #25

Not quite so disastrous, but still pretty dumb. I was logged in over VPN to a client machine. I suspected the problem I was fixing was something to do with the network misbehaving. It was a Windows machine. I thought "I know: I'll disable and reenable the network device!". A few moments later: "Oh. Bugger.". Shortly followed by an embarrassing call to the client's IT department to ask them to reenable the network dev…

Yeah, this is common. Don't modify firewall rules, fuck with interface settings and addressing without another way in.
Post reply on HN