Any manager who doesn't take responsibility for this isn't a manager you'd want to work for. The manager should be fired.
How I Fired Myself
11–20 of 424 posts
Re: How I Fired Myself
#12Wow, I'm sorry you had to experience that. I'm sure it was traumatic -- or perhaps you took it better than I would have. It must be of some comfort to look back now and realize that you only bore a small part of the blame, and that ultimately a large potion of the responsibility lies on the shoulders of whomever set up the dev environment like that, as well as whomever cancelled backups.
Re: How I Fired Myself
#13Whoever cancelled the backups was equally responsible
More responsible, I would say. You expect a junior to make mistakes; the company should be structured to handle that happening. Though I would look askance at whoever hired a philosophy grad as well, to be perfectly honest. The author admits he didn't have the experience to spot bad practice at the time.
You don't expect a junior to make mistakes, you expect everyone AND everything to make mistakes/fail. Backups and separated environments is the least you could expect from a company earning millions.
Re: How I Fired Myself
#14Whoever cancelled the backups was equally responsible
More responsible, I would say. You expect a junior to make mistakes; the company should be structured to handle that happening. Though I would look askance at whoever hired a philosophy grad as well, to be perfectly honest. The author admits he didn't have the experience to spot bad practice at the time.
[deleted]
Re: How I Fired Myself
#15This post just made me feel 10x smarter (not that I blame the author - the blame here lies at the feet of the "Senior" Devs).
Re: How I Fired Myself
#16This is certainly a monumental fuckup, but these things inevitably happen even with better development practices, this is why you need backups, preferably daily, and as much separation of concerns and responsibilities as humanly possible.
Anecdote:
I am working for a company that does some data analysis for marketers aggregated from a vast number of sources. There was a giant legacy MyISAM (this becomes important later) table with lots of imported data. One day, I made some trivial looking migration (added a flag column to that table). I tested it locally, rolled it out to staging server. Everything seemed A-OK until we started migration on the production server. Suddenly, everything broke. By everything, I mean EVERYTHING, our web application showed massive 500-s, total DEFCON1 across the whole company. It turned out we ran out of disk space, since apparently myisam tables are altered the following way: first the table is created with updated schema, then it is populated with data from the old table. MyISAM ran out of disk space and somehow corrupted the existing tables, mysql server would start with blank tables, with all data lost.
I can confirm this very feeling: "The implications of what I'd just done didn't immediately hit me. I first had a truly out-of-body experience, seeming to hover above the darkened room of hackers, each hunched over glowing terminals." Also, I distinctly remember how I shivered and my hands shook. It felt like my body temperature fell by several degrees.
Fortunately for me, there was a daily backup routine in place. Still, several hour long outage and lots of apologies to angry clients.
"There are two types of people in this world, those who have lost data, and those who are going to lose data"
Re: How I Fired Myself
#17I still remember the all-consuming dread I felt as an intern when I ran an UPDATE and forgot the WHERE clause. I consider it part of the rite of passage in the web developer world. Kind of like using an image or text in a dev environment that you never expect a client to see.
Luckily the company I was at (like any rational company) backed up their db and worked in different environments, so it was more of a thing my coworkers teased me for than an apocalyptic event.
Re: How I Fired Myself
#18I know how you felt. Many years ago when I was a junior working in a casual game company, I were to add a bunch of credits to a poker player (fake money). I forget the where in the SQL clause and added credits to every player in our database. Lucky me it was an add and not a set and I could revert it. Another time I was going to shutdown my pc (a debian box) using "shutdown -h now" and totally forgot that I was in a ssh session to our main game server. I had to call the tech support overseas and tell him to physically turn on the server...
Re: How I Fired Myself
#19One of the things I like asking candidates is "Tell me about a time you screwed up so royally that you were sure you were getting fired."
Let's be honest, we all have one or two.. and if you don't, then your one or two are coming. It's what you learned to do differently that I care about.
And if you don't have one, you're either a) incredibly lucky, b) too new to the industry, or c) lying.
Re: How I Fired Myself
#20Wow, that's terrible. Mistakes happen, and for the notion of 'blame' to surface requires some monumentally incompetent management... the exact kind that would have their junior programmers developing against an un-backed-up production database.
The immediate takeaway from a disaster should always be 'How can we make sure this doesn't happen again?' not 'Man, I can't believe Fred did that, what an idiot.'