Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Times I've Messed Up as a Developer

medium.com

Post reply on HN
Print view View original

61–70 of 137 posts

Re: Times I've Messed Up as a Developer

#61

Post by dasil003 »

stephengillie wrote: post #59
avereveard wrote: post #35

Earlier quoted context omitted.

Two terms was such a common source of mistakes: I shut down a server parked in a nearby country and we had to drive there to turn it on. Since then I’ve adopted custom prompt and color scheme for production. I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages

> I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages This is everyday Operations work. There is no school that teaches the eclectic mix of OS and container, networks and routes, app/db/code, CI/CD, BC/DR, writing KB articles, running standups and meetings, triage technical issues, configuration management, Change Control, and Incident Mana…

I also wonder how effective a program can be at teaching this kind of thing. No amount of studying ever ingrained something in my mind as well as the adrenaline shot of somehow fucking up production. I'm not aware of any pedagogical methods to engage the endocrine system, but I find it very useful for retention.

Re: Times I've Messed Up as a Developer

#62

Post by craftyguy »

madamelic wrote: post #51

Here is a fun one for you: In Git, a push-force used to (or at least how I remember it) only force-push your current branch. I force-pushed on a branch... and it force-pushed master. Thankfully no one had pushed to master since I last pulled from it. Here is the fix: ``` [push] default = current ``` Or you can just, not force-push... or be explicit, which I do now. I use force pushes for rebasing and --amend. I am on…

I guess we're both weirdos, since that's exactly what I do. I made it a habit to be explicit when doing the force-push (e.g. git push --force origin ).

Re: Times I've Messed Up as a Developer

#63

Post by 3131s »

seanwilson wrote: post #41
1ba9115454 wrote: post #2

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

This is why I do a "mv" to "tmp" instead of using "rm" when I can. Having special messages or colours to differentiate between production and other environments is a great idea as well. Generally if I'm about to do something on production, I'll close everything else to do with other environments as it's just too easy to slip up.

You can also alias "rm -i" or "rm -I" to "rm" to avert major catastrophes.

From man rm:

-i prompt before every removal

-I prompt once before removing more than three files, or when removing recursively; less intrusive than -i, while still giving protection against most mistakes

Re: Times I've Messed Up as a Developer

#64

Post by bg4 »

brianwawok wrote: post #39
bg4 wrote: post #28

Stop mucking around in production. Stop. Seriously, stop.

If you have never screwed up anything in production, no one at your org trusts you enough to give you production access. Which is ok and great for you, but at the end of the day SOMEONE has to have production access. You can write a devops template that wipes out data just as easy as you can sudo rm -rf /

Imo, it's not about trust, it's about rigor.

Re: Times I've Messed Up as a Developer

#65

Post by cafard »

leejo wrote: post #3

I've seen a DBA drop a production table as well, fortunately it was a small in memory table for non persisting data that could be quickly repopulated. Still lead to about 5 mins of downtime. I always have two terminals open, both running tmux. One is white text on black for dev work, the other is white text on red. I only ever ssh into production servers on the white on red terminal so there's a strong visual signal…

I once got a call from someone with DBA privileges: "I just ran 'SELECT table_name FROM user_tables' and it says 'no rows selected." I logged in and checked, and said, 'Yep, you have no tables.' Fortunately, Oracle by then had the RECYCLE$BIN, and the interruption in service was minimal. I'm not sure how he dropped all the tables.

And let's not talk about my own (mostly minor) disasters.

Re: Times I've Messed Up as a Developer

#67

Post by busterarm »

mfontani wrote: post #57
1ba9115454 wrote: post #2

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

To prevent this, my iTerm2 has a custom background enabled when logged in on any production servers. I wish the same were easily possible on a Linux terminal without much "trickery" :|

I have a different 'theme' for tmux on any production servers I set up than development/test servers and my local machine. It ends up that I nest my remote sessions in my local, but it's effective.

Re: Times I've Messed Up as a Developer

#68

Post by lovehashbrowns »

gumby wrote: post #53
avereveard wrote: post #35

Earlier quoted context omitted.

Two terms was such a common source of mistakes: I shut down a server parked in a nearby country and we had to drive there to turn it on. Since then I’ve adopted custom prompt and color scheme for production. I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages

> Since then I’ve adopted custom prompt and color scheme for production. An AHA moment for me. I've always wondered why people cared about all the terminal customization options (I grew up typing on an 80x48 AAA glass TTY). This is a good reason. I've used a custom prompt with the host name built in for decades, but this kind of color issue makes sense!

I always have so many terminals open that this is a must for me. I have different colors for production and I used ansible to make them consistent. It's nice to not have to waste brain power trying to guess which terminal I'm on.

Re: Times I've Messed Up as a Developer

#69

Post by brianwawok »

bg4 wrote: post #64
brianwawok wrote: post #39

Earlier quoted context omitted.

If you have never screwed up anything in production, no one at your org trusts you enough to give you production access. Which is ok and great for you, but at the end of the day SOMEONE has to have production access. You can write a devops template that wipes out data just as easy as you can sudo rm -rf /

Imo, it's not about trust, it's about rigor.

Yah? Good luck with that

Re: Times I've Messed Up as a Developer

#70

Post by msangi »

Once I refactored some stringly typed C# code to use an enum with values Prod and Test instead of the two strings "Prod" and "Test".

I was relying on expressions like

  mode == "Test"
to fail to type-check to spot all the places that I needed to update.

Sadly some parts of the codebase weren't very idiomatic and had some

  "Test".Equals(mode)
This caused a bit of involuntary testing in production and thought me not to trust anything while refactoring.
Post reply on HN