Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Times I've Messed Up as a Developer

medium.com

Post reply on HN
Print view View original

41–50 of 137 posts

Re: Times I've Messed Up as a Developer

#41

Post by seanwilson »

1ba9115454 wrote: post #2

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

This is why I do a "mv" to "tmp" instead of using "rm" when I can. Having special messages or colours to differentiate between production and other environments is a great idea as well. Generally if I'm about to do something on production, I'll close everything else to do with other environments as it's just too easy to slip up.

Re: Times I've Messed Up as a Developer

#42

Post by dkersten »

1ba9115454 wrote: post #2

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

I used to setup my shell so that the text on the production servers was red and local it was white. Made it easy to know which machine I was tying into :)

Re: Times I've Messed Up as a Developer

#43

Post by philipov »

e12e wrote: post #24
philipov wrote: post #6

Earlier quoted context omitted.

I used to do the same thing with RDPs to windows production servers. I'd set up a different colored color scheme on each computer, with prod being bright red :)

For Windows work, I made sure to have one regular user and one admin user. And set the desktop background of the admin user to red. An additional benefit is that the garish background is uncomfortable to work with, and so discourages being logged in as admin all the time...

Changing the desktop background alone isn't enough. You will inevitably get into a situation where fullscreen windows are occluding it. To avoid any ambiguity, set the color scheme to high contrast and then set all the windows title bars to red, and tint the task bar red. On QA, I tint everything purple, and on DR I tint everything green, or yellow.

Re: Times I've Messed Up as a Developer

#44

Post by mrsernine »

That time I had to update the users table so users belonging to 'department A' should now belong to 'department B'. I wrote that really simple update sentence and even diligently tested it on one of the pre-production servers. Then I copied & pasted it into the production console just to notice that I hadn't copied the WHERE clause of the sentence and now all of the users belonged to the same department.

The fix was not painful, just restore de user table from last night's backup. but I felt a little ashamed when i had to explain the database administrator what has happened.

From that day, I keep autocommit always off.

Re: Times I've Messed Up as a Developer

#45

Post by optimuspaul »

brianwawok wrote: post #39
bg4 wrote: post #28

Stop mucking around in production. Stop. Seriously, stop.

If you have never screwed up anything in production, no one at your org trusts you enough to give you production access. Which is ok and great for you, but at the end of the day SOMEONE has to have production access. You can write a devops template that wipes out data just as easy as you can sudo rm -rf /

I say that all the time. You automation can be just as dangerous as prod ssh access.

Re: Times I've Messed Up as a Developer

#46

Post by megaman22 »

brianwawok wrote: post #34
charlesdm wrote: post #19

Earlier quoted context omitted.

Well played

The flip side to the story.. what if he messed up the restore, and instead of restoring from the only known backup, he wiped our the only known backup? Would be epic fired story. I vote for admit when you mess up, and ask for help. Don't be too proud, we all mess up.

I wonder if kids these days manage to get as much experience recovering a borked computer? I vaguely remember poking at autoexec.bat and config.sys files to get games working, or to attempt to revert some screw-up when I failed to get one working. Or becoming very good at using Linux live CDs to grab personal docs/tax information/business slips... off a hard-disk before wiping it and reinstalling Windows, after a bad driver update or dodgy install of some software.

The first few times, you get that icy "oh shit" feeling, but figuring out how to fix it, and that it can (almost) always be fixed, is invaluable.

Re: Times I've Messed Up as a Developer

#47

Post by mason55 »

dkersten wrote: post #42
1ba9115454 wrote: post #2

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

I used to setup my shell so that the text on the production servers was red and local it was white. Made it easy to know which machine I was tying into :)

> I used to setup my shell so that the text on the production servers was red

I do the same. People always walk up and make some snarky comment like "wow, nice background color" and then I explain why I have a red terminal background and they say "oh... that's a great idea"

Re: Times I've Messed Up as a Developer

#48

Post by madamelic »

w0m wrote: post #21

At my first job, guy in my group once su'd to system user that ran our infrastructure; and typed, cd && rm -rf * and went home. I was sitting at my desk watching service after service slowly disappear in utter confusion, then hours spent cleaning...

... Why would someone do that? Was it ineptness? Just being tired? That seems so blatant.

Re: Times I've Messed Up as a Developer

#49

Post by madamelic »

brianwawok wrote: post #39
bg4 wrote: post #28

Stop mucking around in production. Stop. Seriously, stop.

If you have never screwed up anything in production, no one at your org trusts you enough to give you production access. Which is ok and great for you, but at the end of the day SOMEONE has to have production access. You can write a devops template that wipes out data just as easy as you can sudo rm -rf /

I am much more terrified of Terraform than myself, especially `terraform destroy`. I mean, a human can only do so much so quickly but a script could kill everything semi-instantly.

I look at `terraform plan -destroy` and it still terrifies to destroy an internal service.

Re: Times I've Messed Up as a Developer

#50

Post by benburleson »

My big whoops was remotely bricking a server on top of Mt Haleakala.

I needed to update MySQL to take advantage of some new feature that young, eager me just _needed_ to use. Well, the package repo didn't have that version, so no worries, I'll install from source. Well that required some newer version of a core utility, so no worries, I'll just `yum update` that. No dice. No worries, I'll just force remove and install the newer one.

Well, young, eager me didn't realize essentially every command relied on that core utility, so although I had a prompt, I couldn't even `ls`, much less `yum install`.

Our site technician had to take an OS disk up with him next time he made the 2+ hour drive to the top. Luckily we did have a KVM set up so it didn't require a site visit from me (or is that unlucky...it would have been a trip to Maui!).

Post reply on HN