Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Times I've Messed Up as a Developer

medium.com

Post reply on HN
Print view View original

71–80 of 137 posts

Re: Times I've Messed Up as a Developer

#71

Post by humblebee »

1ba9115454 wrote: post #2

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

One the last day of a colleague of mine, he decided to run `rm -rf --no-preserve-root /` to see what actually would happen on his workstation (it was going to be formatted shortly after). We were all standing around talking waiting for whatever would happen to happen before we saw him off. A few minutes in I asked if he had unmounted our NFS volumes. Nope. He quickly pulled the power, and that was that. I have no idea if anything was deleted, but he probably had ~ 2PB of mounted data attached to his workstation. Everything from build archives, customer logs dumps, test artifacts, dogfooding clusters (the product was a parallel distributed networked file system), and possibly a few random other things.

No idea if anything would have happened, most of the data was read only (builds, logs, etc) but some was not.

Re: Times I've Messed Up as a Developer

#72

Post by iza »

madamelic wrote: post #51

Here is a fun one for you: In Git, a push-force used to (or at least how I remember it) only force-push your current branch. I force-pushed on a branch... and it force-pushed master. Thankfully no one had pushed to master since I last pulled from it. Here is the fix: ``` [push] default = current ``` Or you can just, not force-push... or be explicit, which I do now. I use force pushes for rebasing and --amend. I am on…

In Git 2.0 they changed the default push mode to "simple" which prevents this issue.

Re: Times I've Messed Up as a Developer

#73

Post by kbenson »

humblebee wrote: post #71
1ba9115454 wrote: post #2

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

One the last day of a colleague of mine, he decided to run `rm -rf --no-preserve-root /` to see what actually would happen on his workstation (it was going to be formatted shortly after). We were all standing around talking waiting for whatever would happen to happen before we saw him off. A few minutes in I asked if he had unmounted our NFS volumes. Nope. He quickly pulled the power, and that was that. I have no ide…

This is why --one-file-system should be default, with a warning if another file system is encountered that's skipped so if it's what is desired you can re-run it (or umount and try again).

Re: Times I've Messed Up as a Developer

#74

Post by mmjaa »

mfontani wrote: post #57
1ba9115454 wrote: post #2

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

To prevent this, my iTerm2 has a custom background enabled when logged in on any production servers. I wish the same were easily possible on a Linux terminal without much "trickery" :|

Whats wrong with just setting PS1 to indicate that its a production server, and using a decent host naming scheme that properly indicates the function of the machine you're logged into? I mean this is what I do for my Linux machines, and I haven't deleted a Linux production machine in decades ..

Re: Times I've Messed Up as a Developer

#76

Post by kbenson »

dasil003 wrote: post #61
stephengillie wrote: post #59

Earlier quoted context omitted.

> I really wish there was some practical information course at compsci with these kind of tips to prevent the most common slippages This is everyday Operations work. There is no school that teaches the eclectic mix of OS and container, networks and routes, app/db/code, CI/CD, BC/DR, writing KB articles, running standups and meetings, triage technical issues, configuration management, Change Control, and Incident Mana…

I also wonder how effective a program can be at teaching this kind of thing. No amount of studying ever ingrained something in my mind as well as the adrenaline shot of somehow fucking up production. I'm not aware of any pedagogical methods to engage the endocrine system, but I find it very useful for retention.

Well, part of the learning is making the mistakes yourself, and part of it is your paranoid coworkers cautioning you against certain behavior and relaying the horror stories of what it might result in as experienced by themselves or someone they knew. A thedailywtf.com for operations mishaps would probably go a long way.

Re: Times I've Messed Up as a Developer

#78

Post by woodrowbarlow »

seanwilson wrote: post #41
1ba9115454 wrote: post #2

Back in the early 90's I had 2 terminals up on my X windows console. One for local dev and another for a production server running a trading system for around 50 users. Naturally for production I was logged in as root. I typed rm -rf to clear out my dev folder only to figure out I just deleted the root folder and all sub folders on a production machine. I recalled a piece of advice a university friend had given me. "…

This is why I do a "mv" to "tmp" instead of using "rm" when I can. Having special messages or colours to differentiate between production and other environments is a great idea as well. Generally if I'm about to do something on production, I'll close everything else to do with other environments as it's just too easy to slip up.

my go-to .bashrc hashes the hostname into a color and prints the prompt in that color. it has saved me from making a dumb mistake a few times.

Re: Times I've Messed Up as a Developer

#79

Post by FiveSquared »

megaman22 wrote: post #46
brianwawok wrote: post #34

Earlier quoted context omitted.

The flip side to the story.. what if he messed up the restore, and instead of restoring from the only known backup, he wiped our the only known backup? Would be epic fired story. I vote for admit when you mess up, and ask for help. Don't be too proud, we all mess up.

I wonder if kids these days manage to get as much experience recovering a borked computer? I vaguely remember poking at autoexec.bat and config.sys files to get games working, or to attempt to revert some screw-up when I failed to get one working. Or becoming very good at using Linux live CDs to grab personal docs/tax information/business slips... off a hard-disk before wiping it and reinstalling Windows, after a bad…

Well, I know I how to, and I am fairly young.

Re: Times I've Messed Up as a Developer

#80

Post by Posibyte »

Fortune 500 company, Junior dev. Tried to log into our staging database, which has the same username, but different password than production.

Tried 3 times, got the locked out error. Got frustrated, said I'd fix it after lunch, went out to eat, came back about 3 hours later after a doctor's appointment.

My boss was waiting for me at the department door and I got sent back to his office for a stern talking to. I had locked out the production user from the database, causing every app in the company to go offline. I was then told to read aloud back all of the projected sales losses from that day as well as write a bunch of letters for our top sales people who couldn't make sales that day.

Wasn't fired (thankfully). Made it to a senior lead, so if I had to say anything, it's that the punishment really made it really drive home and I _never ever did it again_ . :)

Post reply on HN