Live data from Hacker News

Times I've Messed Up as a Developer

medium.com

51–60 of 137 posts

Re: Times I've Messed Up as a Developer

#51
Here is a fun one for you:

In Git, a push-force used to (or at least how I remember it) only force-push your current branch. I force-pushed on a branch... and it force-pushed master. Thankfully no one had pushed to master since I last pulled from it.

Here is the fix:

``` [push] default = current ```

Or you can just, not force-push... or be explicit, which I do now. I use force pushes for rebasing and --amend. I am one of those weirdos who enjoys clean, readable git repos.

Re: Times I've Messed Up as a Developer

#52
post #7

I was on my way out the door at my first job, and an SEO person stopped me and asked me if I could fix something. I just needed to delete a single row from a database. Easy enough. I get as far as “DELETE FROM table_name” and for some unknown reason run the damn thing without a WHERE. All the rows, gone! We did have nightly backups but the guy with access to them was notoriously hard to get the attention of and lived…

This 'default' behaviour strikes me as one of SQL's worst faults. If you want to delete all rows, you should be made to explicitly type out that intention.

I'd say require a WHERE on deletes. If you want "WHERE true" that's fine, but require a WHERE.

Re: Times I've Messed Up as a Developer

#53

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

Two terms was such a common source of mistakes: I shut down a server parked in a nearby country and we had to drive there to turn it on. Since then I’ve adopted custom prompt and color scheme for production. I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages

> Since then I’ve adopted custom prompt and color scheme for production.

An AHA moment for me. I've always wondered why people cared about all the terminal customization options (I grew up typing on an 80x48 AAA glass TTY). This is a good reason.

I've used a custom prompt with the host name built in for decades, but this kind of color issue makes sense!

Re: Times I've Messed Up as a Developer

#54

Earlier quoted context omitted.

If you have never screwed up anything in production, no one at your org trusts you enough to give you production access. Which is ok and great for you, but at the end of the day SOMEONE has to have production access. You can write a devops template that wipes out data just as easy as you can sudo rm -rf /

I say that all the time. You automation can be just as dangerous as prod ssh access.

And in some ways it is worse..

I remember at least one prod incident where a script made it out into prod and wiped out 100s of servers. Something like it had rm -rf /$SOME_ENV_VAR which was set in QA but not prod. Opps....

Re: Times I've Messed Up as a Developer

#55
Back in the nineties, at a previous employer, I wrote a code generator that read a kind of DSL from stdin and spat out C source to stdout.

Having written the initial version and got it to compile, I thought I should test it before checking it into RCS for the first time. So I ran it with some simple input and piped the output to something like program.c. At that point I remembered that the code generator's main source code file was also named program.c. To my horror it had overwritten a large part of its own source.

I remember staying in the office until about 2am to re-write the lost code, but my boss never found out.

Re: Times I've Messed Up as a Developer

#56
post #47

Earlier quoted context omitted.

I used to setup my shell so that the text on the production servers was red and local it was white. Made it easy to know which machine I was tying into :)

> I used to setup my shell so that the text on the production servers was red I do the same. People always walk up and make some snarky comment like "wow, nice background color" and then I explain why I have a red terminal background and they say "oh... that's a great idea"

I tried this for a while after a goof up. However because my dev environments more or less looked like default, if I forgot to setup a new production server to change the color, I'd mistake it for local at some point.

So I change my dev to be the non-default now. If I see the default/typical color scheme, I assume production.

Re: Times I've Messed Up as a Developer

#57

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

To prevent this, my iTerm2 has a custom background enabled when logged in on any production servers. I wish the same were easily possible on a Linux terminal without much "trickery" :|

Re: Times I've Messed Up as a Developer

#58
Late 90's, I'm lead dev for a large site. Company hires a DBA. We partner with a VERY large content provider. I'm awake for 48 hours buttoning things up, last minute stuff. I finally collapse from exhaustion only to be woken up at 7am on - go live day.

Nothing works.

A few moments debugging reveals that all the column names in a key set of our datasets have been renamed. No time to waste, I rename them all back as quickly as possible. We go live, no one is the wiser.

We contact the DBA and explain the folly of his decision.

He then proceeds to rename everything AGAIN on a live system. No we didn't shoot him lengthwise, but we did fire him with some prejudice.

Re: Times I've Messed Up as a Developer

#59

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

Two terms was such a common source of mistakes: I shut down a server parked in a nearby country and we had to drive there to turn it on. Since then I’ve adopted custom prompt and color scheme for production. I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages

> I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages

This is everyday Operations work. There is no school that teaches the eclectic mix of OS and container, networks and routes, app/db/code, CI/CD, BC/DR, writing KB articles, running standups and meetings, triage technical issues, configuration management, Change Control, and Incident Management.

These diverse skill sets are often called "Devops", but nobody makes a Devops degree. The only way to assemble this experience is while working to support developers.

Re: Times I've Messed Up as a Developer

#60
post #47

Earlier quoted context omitted.

> I used to setup my shell so that the text on the production servers was red I do the same. People always walk up and make some snarky comment like "wow, nice background color" and then I explain why I have a red terminal background and they say "oh... that's a great idea"

I tried this for a while after a goof up. However because my dev environments more or less looked like default, if I forgot to setup a new production server to change the color, I'd mistake it for local at some point. So I change my dev to be the non-default now. If I see the default/typical color scheme, I assume production.

That’s a pretty good idea. “Be careful” is a good default.
Post reply on HN